2024-03-25T18:22:14.467 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T18:22:14.467 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331 --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331/orig.config.yaml 2024-03-25T18:22:14.507 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:22:14.592 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621331' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBLuZd9QyE9I0h1TLT3GEMuC6haZ0Y7lzY5OJwwzJRByM7qKFM0qvQe2PpjZotyn3JxWXAqMDyg7SYU3euO/Z1k= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNImLuourlybiOWkqsOsHAugVHoQs9Bh5WVVf1tuCtJ0N1zw/lTL5XFajoWPCm4o+1QJQL1y04HdVYF9xa+skvU= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T18:22:14.592 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T18:22:14.593 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T18:22:14.593 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T18:22:14.594 INFO:teuthology.task.internal:Checking packages... 2024-03-25T18:22:14.615 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T18:22:14.615 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T18:22:14.615 INFO:teuthology.packaging:ref: None 2024-03-25T18:22:14.615 INFO:teuthology.packaging:tag: None 2024-03-25T18:22:14.615 INFO:teuthology.packaging:branch: main 2024-03-25T18:22:14.615 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:22:14.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T18:22:14.958 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T18:22:14.960 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T18:22:14.969 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T18:22:14.969 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T18:22:14.998 INFO:teuthology.task.internal:Saving configuration 2024-03-25T18:22:15.013 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T18:22:15.030 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:22:15.056 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:13.883157', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBLuZd9QyE9I0h1TLT3GEMuC6haZ0Y7lzY5OJwwzJRByM7qKFM0qvQe2PpjZotyn3JxWXAqMDyg7SYU3euO/Z1k='} 2024-03-25T18:22:15.079 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:13.882000', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNImLuourlybiOWkqsOsHAugVHoQs9Bh5WVVf1tuCtJ0N1zw/lTL5XFajoWPCm4o+1QJQL1y04HdVYF9xa+skvU='} 2024-03-25T18:22:15.079 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T18:22:15.087 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T18:22:15.088 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T18:22:15.088 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T18:22:15.191 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f53796e50d0>, signals=[15]) 2024-03-25T18:22:15.191 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T18:22:15.200 INFO:teuthology.task.internal:Opening connections... 2024-03-25T18:22:15.200 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-03-25T18:22:15.202 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:22:15.294 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-03-25T18:22:15.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:22:15.382 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T18:22:15.391 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-03-25T18:22:15.409 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-03-25T18:22:15.409 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-03-25T18:22:15.464 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:22:15.465 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:22:15.466 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-03-25T18:22:15.679 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-03-25T18:22:15.696 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-03-25T18:22:15.696 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-03-25T18:22:15.753 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-03-25T18:22:15.753 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9" 2024-03-25T18:22:15.753 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-03-25T18:22:15.753 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:22:15.753 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9" 2024-03-25T18:22:15.753 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:22:15.753 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:22:15.753 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-03-25T18:22:15.753 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:22:15.754 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:22:15.754 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:22:15.754 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:22:15.754 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:22:15.754 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:22:15.754 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-03-25T18:22:15.775 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T18:22:15.785 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T18:22:15.822 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T18:22:15.822 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:22:15.825 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:22:15.841 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T18:22:15.852 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T18:22:15.852 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:22:15.880 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:22:15.985 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T18:22:16.049 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T18:22:16.068 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T18:22:16.068 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T18:22:16.068 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:22:16.069 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:22:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:16.085 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-03-25T18:22:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:16.086 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-03-25T18:22:16.142 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:22:16.142 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.14.0-361.el9.x86_64 2024-03-25T18:22:16.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-03-25T18:22:16.143 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:22:16.143 INFO:teuthology.task.kernel:Running kernel on smithi195: 5.14.0-361.el9.x86_64 2024-03-25T18:22:16.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2024-03-25T18:22:18.217 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T18:22:18.349 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T18:22:19.800 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-25T18:22:19.909 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-25T18:22:23.794 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-25T18:22:23.963 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-25T18:22:28.563 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 19 MB 00:03 2024-03-25T18:22:29.116 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 19 MB 00:04 2024-03-25T18:22:35.716 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-25T18:22:37.254 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-03-25T18:22:38.045 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 1.1 MB/s | 6.1 MB 00:05 2024-03-25T18:22:39.397 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-25T18:22:40.356 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-25T18:22:41.132 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 31 MB/s | 21 MB 00:00 2024-03-25T18:22:44.329 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-25T18:22:46.283 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-25T18:22:46.428 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:22:46.428 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:22:46.551 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:22:46.558 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:22:46.558 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-03-25T18:22:46.558 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:22:46.558 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:22:46.558 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:22:46.558 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-25T18:22:46.558 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:22:46.558 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:22:46.558 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:22:46.558 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:22:46.559 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:22:46.559 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:22:46.559 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2024-03-25T18:22:46.559 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:22:46.559 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 76 M 2024-03-25T18:22:46.559 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 123 M 2024-03-25T18:22:46.559 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:22:46.701 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:46.703 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:46.715 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:46.723 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:46.723 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:46.732 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:46.787 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:46.788 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:46.796 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:46.923 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:46.933 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:46.953 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:46.955 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:46.958 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:47.018 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:47.020 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:47.055 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:47.074 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:47.086 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:47.143 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:47.144 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:47.263 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:47.272 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:47.294 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:47.298 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:47.307 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:47.329 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:47.459 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:47.466 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:47.492 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:47.807 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:47.832 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:47.893 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:47.963 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:47.970 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:48.038 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:48.045 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:48.098 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:48.147 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 43 kB/s | 66 kB 00:01 2024-03-25T18:22:48.168 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:48.204 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:48.222 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:48.404 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:48.423 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:22:48.423 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:22:48.522 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:22:48.538 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:22:48.538 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-03-25T18:22:48.538 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:22:48.538 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:22:48.538 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:22:48.538 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-25T18:22:48.538 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:22:48.539 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:22:48.539 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:22:48.539 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:22:48.539 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:22:48.539 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:22:48.539 INFO:teuthology.orchestra.run.smithi195.stdout:Install 4 Packages 2024-03-25T18:22:48.539 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:22:48.539 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 76 M 2024-03-25T18:22:48.540 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 123 M 2024-03-25T18:22:48.540 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:22:48.589 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 7.6 MB/s | 15 MB 00:01 2024-03-25T18:22:48.623 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:48.665 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:48.665 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:48.666 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:48.675 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:48.692 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:48.810 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:48.821 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:48.821 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:48.845 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:48.884 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:48.894 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:48.896 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:48.922 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:48.992 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:49.027 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:49.049 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:49.069 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:49.097 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:49.213 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:49.233 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:49.262 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:49.414 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 12 MB/s | 33 MB 00:02 2024-03-25T18:22:49.472 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:49.646 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:49.670 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:49.682 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:49.700 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:49.705 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:49.717 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:49.720 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:49.729 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:49.757 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:49.816 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:49.825 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:49.918 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:50.000 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:50.014 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:50.082 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:50.083 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:50.092 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:50.155 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:50.164 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:50.173 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:50.181 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:50.189 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:50.231 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 13 MB/s | 27 MB 00:02 2024-03-25T18:22:50.236 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:50.236 INFO:teuthology.orchestra.run.smithi107.stdout:Total 21 MB/s | 76 MB 00:03 2024-03-25T18:22:50.279 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:50.297 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:50.442 INFO:teuthology.orchestra.run.smithi195.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 36 kB/s | 66 kB 00:01 2024-03-25T18:22:50.677 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:50.899 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:22:50.968 INFO:teuthology.orchestra.run.smithi195.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 6.3 MB/s | 15 MB 00:02 2024-03-25T18:22:50.971 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:51.121 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:51.208 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:51.286 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:22:51.286 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:22:51.526 INFO:teuthology.orchestra.run.smithi195.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 11 MB/s | 33 MB 00:02 2024-03-25T18:22:51.528 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:51.688 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:51.705 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:51.723 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:51.885 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:51.941 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:51.996 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:52.292 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:22:52.293 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:22:52.751 INFO:teuthology.orchestra.run.smithi195.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 12 MB/s | 27 MB 00:02 2024-03-25T18:22:52.756 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:52.756 INFO:teuthology.orchestra.run.smithi195.stdout:Total 18 MB/s | 76 MB 00:04 2024-03-25T18:22:53.423 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:22:53.786 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:22:53.787 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:22:54.770 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:22:54.795 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:22:54.795 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:22:55.388 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:22:55.489 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:22:57.857 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:22:57.944 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:58.221 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:58.366 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:00.696 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:00.838 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:03.048 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:03.091 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:05.518 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:05.630 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:07.844 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:10.333 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:37.072 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:37.072 INFO:teuthology.orchestra.run.smithi107.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:23:37.072 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:23:37.117 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:37.465 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:37.465 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:23:37.465 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:37.465 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:37.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:37.883 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:23:37.883 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:23:37.883 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:37.883 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:37.884 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:23:37.884 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:37.884 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:23:37.884 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:23:38.308 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:38.407 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:38.407 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:38.407 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:23:38.407 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:23:38.407 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:23:38.408 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T18:23:38.408 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:23:38.408 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:38.486 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:38.487 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:38.487 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-03-25T18:23:38.521 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-03-25T18:23:38.521 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.06 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 68.el9 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Group : Unspecified 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 8264254 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : builder@centos.org 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-03-25T18:23:38.522 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:23:38.523 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-03-25T18:23:38.523 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-03-25T18:23:38.523 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:23:38.523 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:23:38.523 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:23:38.523 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:23:38.523 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-03-25T18:23:38.523 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:23:38.523 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:23:38.524 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:23:38.524 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:23:38.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:23:39.117 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-03-25T18:23:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:39.555 INFO:teuthology.orchestra.run.smithi195.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:23:39.555 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:23:39.599 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:39.988 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:39.988 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:23:39.988 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:39.988 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:40.389 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:40.389 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:23:40.389 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:23:40.389 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:40.389 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:40.389 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:23:40.389 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:40.389 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:23:40.389 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:23:40.440 INFO:teuthology.orchestra.run.smithi107.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:23:40.477 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-03-25T18:23:40.479 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-03-25T18:23:40.494 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.M2DylyOSKf 2024-03-25T18:23:40.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.M2DylyOSKf 2024-03-25T18:23:40.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.M2DylyOSKf 2024-03-25T18:23:40.697 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.M2DylyOSKf is 6KB 2024-03-25T18:23:40.699 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:40.746 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.M2DylyOSKf 2024-03-25T18:23:40.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:23:40.801 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:40.801 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:40.801 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:23:40.801 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:23:40.801 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:23:40.801 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T18:23:40.801 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:23:40.802 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:40.824 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:23:40.824 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:23:40.824 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:23:40.824 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:23:40.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:23:40.881 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:40.881 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:40.881 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2024-03-25T18:23:40.916 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2024-03-25T18:23:40.916 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2024-03-25T18:23:40.916 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.06 2024-03-25T18:23:40.916 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 68.el9 2024-03-25T18:23:40.916 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2024-03-25T18:23:40.916 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:23:40.916 INFO:teuthology.orchestra.run.smithi195.stdout:Group : Unspecified 2024-03-25T18:23:40.916 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 8264254 2024-03-25T18:23:40.916 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2024-03-25T18:23:40.916 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : builder@centos.org 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:23:40.917 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:23:40.918 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2024-03-25T18:23:40.918 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:23:40.918 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:23:40.918 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:23:40.919 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:23:40.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:23:40.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-03-25T18:23:41.505 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2024-03-25T18:23:42.851 INFO:teuthology.orchestra.run.smithi195.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:23:42.871 INFO:teuthology.orchestra.run.smithi195.stderr:done 2024-03-25T18:23:42.874 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-03-25T18:23:42.889 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.lJr3Tz0wPR 2024-03-25T18:23:42.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lJr3Tz0wPR 2024-03-25T18:23:42.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.lJr3Tz0wPR 2024-03-25T18:23:43.104 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.lJr3Tz0wPR is 6KB 2024-03-25T18:23:43.152 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.lJr3Tz0wPR 2024-03-25T18:23:43.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:23:43.236 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:23:43.236 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:23:43.236 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:23:43.236 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:23:43.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:23:43.394 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2024-03-25T18:24:10.975 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T18:24:10.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-25T18:24:10.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:13.400 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T18:24:13.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-25T18:24:13.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:29.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-03-25T18:24:31.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-03-25T18:24:38.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-25T18:24:38.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:40.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-25T18:24:40.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:41.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-03-25T18:24:53.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-25T18:24:53.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:11.967 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-03-25T18:25:26.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-25T18:25:26.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:27.426 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-03-25T18:25:27.968 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-25T18:25:27.968 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:25:27.968 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-03-25T18:25:28.023 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:25:28.023 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:25:28.023 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:25:28.023 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:25:29.024 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T18:25:29.024 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:25:29.095 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-03-25T18:25:29.146 DEBUG:teuthology.parallel:result is None 2024-03-25T18:25:40.780 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T18:25:52.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-25T18:25:52.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:53.177 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-03-25T18:25:53.577 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-25T18:25:53.577 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:25:53.577 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-03-25T18:25:53.593 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:25:53.593 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:25:53.593 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:25:53.593 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:25:54.594 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T18:25:54.594 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:25:54.669 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2024-03-25T18:25:54.702 DEBUG:teuthology.parallel:result is None 2024-03-25T18:25:54.703 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T18:25:54.710 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T18:25:54.710 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:25:54.713 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:25:54.731 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T18:25:54.737 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T18:25:54.776 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T18:25:54.776 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:25:54.779 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:25:54.806 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T18:25:54.818 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T18:25:54.818 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:25:54.842 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:25:54.879 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:54.879 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:54.893 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:54.893 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:54.895 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T18:25:54.902 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T18:25:54.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:25:54.938 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:25:54.967 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T18:25:54.976 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T18:25:54.977 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:25:55.009 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:25:55.025 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:55.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:55.157 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:55.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:55.280 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T18:25:55.281 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:25:55.347 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:55.392 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:55.442 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:55.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:55.569 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T18:25:55.569 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:25:55.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-03-25T18:25:55.640 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-03-25T18:25:55.697 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:25:55.726 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:25:56.175 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T18:25:56.183 INFO:teuthology.task.internal:Starting timer... 2024-03-25T18:25:56.183 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T18:25:56.230 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T18:25:56.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-03-25T18:25:56.345 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-03-25T18:25:56.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2024-03-25T18:25:56.401 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2024-03-25T18:25:56.403 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T18:25:56.403 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-03-25T18:25:56.422 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-03-25T18:25:56.422 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2024-03-25T18:25:56.447 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2024-03-25T18:25:56.447 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2024-03-25T18:25:56.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:25:56.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:25:56.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:25:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:25:56.521 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T18:25:56.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:25:56.566 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:25:56.607 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T18:25:56.616 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T18:25:56.631 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T18:25:56.633 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:30:12.980 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-03-25T18:30:12.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-25T18:30:12.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:30:13.078 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-03-25T18:30:13.161 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-25T18:30:13.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-25T18:30:13.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:30:13.252 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-03-25T18:30:13.330 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-25T18:30:13.331 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:30:13.343 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:30:13.343 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:30:13.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:30:13.346 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:30:13.347 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:30:13.377 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:30:13.394 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:30:13.403 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:30:13.421 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:30:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-03-25T18:30:13.437 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-03-25T18:30:13.450 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2024-03-25T18:30:13.453 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:30:13.464 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2024-03-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:30:13.481 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:30:13.497 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:30:13.518 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:30:13.521 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:30:13.521 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-03-25T18:30:13.521 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:13.522 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:13.522 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:13.522 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:13.553 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:30:13.557 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:30:13.557 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-03-25T18:30:13.557 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:13.557 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:13.557 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:13.557 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:13.557 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:30:13.565 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:30:13.566 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:30:13.566 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:30:13.566 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:30:13.573 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:30:13.573 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:30:13.574 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:30:13.574 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:30:13.577 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:30:13.577 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:30:13.827 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:30:13.827 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:30:13.834 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:30:13.834 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:30:14.174 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:30:14.174 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T18:30:14.174 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:30:14.203 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:30:14.203 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:30:14.249 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:30:14.249 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T18:30:14.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:30:14.272 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:30:14.279 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:30:14.279 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:30:14.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:30:14.352 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:30:14.419 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2024-03-25T18:30:14.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2024-03-25T18:30:14.428 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:30:14.495 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-03-25T18:30:14.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-03-25T18:30:14.654 INFO:teuthology.orchestra.run.smithi195.stdout:74 files removed 2024-03-25T18:30:14.681 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw 2024-03-25T18:30:14.748 INFO:teuthology.orchestra.run.smithi107.stdout:74 files removed 2024-03-25T18:30:14.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2024-03-25T18:30:15.814 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 98 kB/s | 82 kB 00:00 2024-03-25T18:30:16.153 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 76 kB/s | 82 kB 00:01 2024-03-25T18:30:16.673 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T18:30:17.095 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 11 kB/s | 10 kB 00:00 2024-03-25T18:30:17.615 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 2.1 kB/s | 1.9 kB 00:00 2024-03-25T18:30:17.921 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T18:30:18.134 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T18:30:18.473 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T18:30:19.720 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-25T18:30:19.990 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-25T18:30:23.866 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-25T18:30:25.437 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:05 2024-03-25T18:30:29.252 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 19 MB 00:04 2024-03-25T18:30:30.499 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:04 2024-03-25T18:30:37.625 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-25T18:30:38.229 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 1.1 MB/s | 6.1 MB 00:05 2024-03-25T18:30:39.947 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T18:30:40.593 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-25T18:30:41.938 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-25T18:30:42.581 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 21 MB 00:02 2024-03-25T18:30:46.886 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-25T18:30:47.539 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-25T18:30:49.313 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:30:49.339 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:30:49.339 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-25T18:30:49.339 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:30:49.340 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:30:49.341 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout:Installing weak dependencies: 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:30:49.342 INFO:teuthology.orchestra.run.smithi195.stdout:Install 30 Packages 2024-03-25T18:30:49.343 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 2 Packages 2024-03-25T18:30:49.343 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:30:49.343 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 58 M 2024-03-25T18:30:49.343 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:30:49.990 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:30:50.010 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:30:50.011 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:30:50.012 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:30:50.013 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:30:50.014 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-03-25T18:30:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:30:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:30:50.014 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:30:50.014 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:30:50.014 INFO:teuthology.orchestra.run.smithi107.stdout:Install 30 Packages 2024-03-25T18:30:50.014 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2024-03-25T18:30:50.014 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:30:50.015 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 58 M 2024-03-25T18:30:50.015 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:30:59.386 INFO:teuthology.orchestra.run.smithi195.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 567 kB/s | 5.3 MB 00:09 2024-03-25T18:30:59.486 INFO:teuthology.orchestra.run.smithi195.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T18:31:00.828 INFO:teuthology.orchestra.run.smithi195.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 553 kB/s | 742 kB 00:01 2024-03-25T18:31:01.753 INFO:teuthology.orchestra.run.smithi195.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 539 kB/s | 498 kB 00:00 2024-03-25T18:31:03.771 INFO:teuthology.orchestra.run.smithi107.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 404 kB/s | 5.3 MB 00:13 2024-03-25T18:31:03.988 INFO:teuthology.orchestra.run.smithi107.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 116 kB/s | 25 kB 00:00 2024-03-25T18:31:06.529 INFO:teuthology.orchestra.run.smithi107.stdout:(3/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 612 kB/s | 9.7 MB 00:16 2024-03-25T18:31:07.380 INFO:teuthology.orchestra.run.smithi107.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 587 kB/s | 498 kB 00:00 2024-03-25T18:31:07.621 INFO:teuthology.orchestra.run.smithi107.stdout:(5/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 204 kB/s | 742 kB 00:03 2024-03-25T18:31:07.821 INFO:teuthology.orchestra.run.smithi107.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 226 kB/s | 45 kB 00:00 2024-03-25T18:31:08.230 INFO:teuthology.orchestra.run.smithi107.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 322 kB/s | 131 kB 00:00 2024-03-25T18:31:09.197 INFO:teuthology.orchestra.run.smithi107.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 167 kB/s | 162 kB 00:00 2024-03-25T18:31:10.180 INFO:teuthology.orchestra.run.smithi107.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 326 kB/s | 320 kB 00:00 2024-03-25T18:31:10.453 INFO:teuthology.orchestra.run.smithi195.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 600 kB/s | 5.1 MB 00:08 2024-03-25T18:31:10.561 INFO:teuthology.orchestra.run.smithi195.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 417 kB/s | 45 kB 00:00 2024-03-25T18:31:10.811 INFO:teuthology.orchestra.run.smithi195.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 525 kB/s | 131 kB 00:00 2024-03-25T18:31:10.955 INFO:teuthology.orchestra.run.smithi107.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 384 kB/s | 298 kB 00:00 2024-03-25T18:31:11.145 INFO:teuthology.orchestra.run.smithi195.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 486 kB/s | 162 kB 00:00 2024-03-25T18:31:11.247 INFO:teuthology.orchestra.run.smithi107.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 341 kB/s | 99 kB 00:00 2024-03-25T18:31:11.356 INFO:teuthology.orchestra.run.smithi107.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-25T18:31:11.397 INFO:teuthology.orchestra.run.smithi107.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-25T18:31:11.431 INFO:teuthology.orchestra.run.smithi107.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:31:11.490 INFO:teuthology.orchestra.run.smithi107.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:31:11.532 INFO:teuthology.orchestra.run.smithi107.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:31:11.565 INFO:teuthology.orchestra.run.smithi107.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:31:11.704 INFO:teuthology.orchestra.run.smithi195.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 574 kB/s | 320 kB 00:00 2024-03-25T18:31:11.737 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:12.104 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:12.121 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:12.176 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:12.246 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:12.337 INFO:teuthology.orchestra.run.smithi195.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 470 kB/s | 298 kB 00:00 2024-03-25T18:31:12.381 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:12.416 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:12.482 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:12.587 INFO:teuthology.orchestra.run.smithi195.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 397 kB/s | 99 kB 00:00 2024-03-25T18:31:12.608 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:12.750 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:12.751 INFO:teuthology.orchestra.run.smithi195.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 647 kB/s | 106 kB 00:00 2024-03-25T18:31:12.821 INFO:teuthology.orchestra.run.smithi195.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.2 MB/s | 159 kB 00:00 2024-03-25T18:31:12.825 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:12.855 INFO:teuthology.orchestra.run.smithi195.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:31:12.904 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:12.928 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:12.931 INFO:teuthology.orchestra.run.smithi195.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.5 MB/s | 661 kB 00:00 2024-03-25T18:31:12.972 INFO:teuthology.orchestra.run.smithi195.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 282 kB 00:00 2024-03-25T18:31:13.006 INFO:teuthology.orchestra.run.smithi195.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:31:13.049 INFO:teuthology.orchestra.run.smithi107.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 149 kB/s | 221 kB 00:01 2024-03-25T18:31:13.074 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:13.129 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:13.210 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:13.232 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.276 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:13.287 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.318 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:13.353 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:13.370 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:13.399 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:13.402 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:13.434 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:13.435 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:13.452 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.507 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.574 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:13.650 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:13.667 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:13.704 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.721 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.757 INFO:teuthology.orchestra.run.smithi107.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 51 kB/s | 36 kB 00:00 2024-03-25T18:31:13.840 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:13.920 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:13.937 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:13.992 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:14.014 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:14.025 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:14.058 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:14.076 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:14.108 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:14.140 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:14.155 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:14.244 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:14.279 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:14.296 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:14.317 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:14.356 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:14.410 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:14.428 INFO:teuthology.orchestra.run.smithi107.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:14.439 INFO:teuthology.orchestra.run.smithi195.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 154 kB/s | 221 kB 00:01 2024-03-25T18:31:14.466 INFO:teuthology.orchestra.run.smithi107.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 67 kB/s | 47 kB 00:00 2024-03-25T18:31:14.473 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:14.506 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:14.538 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:14.575 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:14.592 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:14.611 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:14.662 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:14.742 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:14.822 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:14.849 INFO:teuthology.orchestra.run.smithi107.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 805 kB/s | 309 kB 00:00 2024-03-25T18:31:14.978 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:15.067 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:15.120 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:15.137 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:15.141 INFO:teuthology.orchestra.run.smithi107.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-25T18:31:15.173 INFO:teuthology.orchestra.run.smithi195.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 49 kB/s | 36 kB 00:00 2024-03-25T18:31:15.208 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:15.208 INFO:teuthology.orchestra.run.smithi107.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 392 kB/s | 26 kB 00:00 2024-03-25T18:31:15.243 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:15.275 INFO:teuthology.orchestra.run.smithi107.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 729 kB/s | 49 kB 00:00 2024-03-25T18:31:15.275 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:15.308 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:15.324 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:15.341 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:15.342 INFO:teuthology.orchestra.run.smithi107.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-25T18:31:15.393 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:15.442 INFO:teuthology.orchestra.run.smithi107.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-25T18:31:15.475 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:15.517 INFO:teuthology.orchestra.run.smithi107.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-25T18:31:15.562 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-25T18:31:15.717 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:15.810 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:15.859 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:15.876 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:15.915 INFO:teuthology.orchestra.run.smithi195.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 64 kB/s | 47 kB 00:00 2024-03-25T18:31:16.140 INFO:teuthology.orchestra.run.smithi195.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-03-25T18:31:16.324 INFO:teuthology.orchestra.run.smithi195.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-03-25T18:31:16.374 INFO:teuthology.orchestra.run.smithi195.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-03-25T18:31:16.441 INFO:teuthology.orchestra.run.smithi195.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-03-25T18:31:16.525 INFO:teuthology.orchestra.run.smithi195.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 807 kB/s | 67 kB 00:00 2024-03-25T18:31:17.183 INFO:teuthology.orchestra.run.smithi195.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-25T18:31:17.242 INFO:teuthology.orchestra.run.smithi195.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-25T18:31:17.384 INFO:teuthology.orchestra.run.smithi195.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-25T18:31:18.293 INFO:teuthology.orchestra.run.smithi107.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 479 kB/s | 5.1 MB 00:10 2024-03-25T18:31:22.075 INFO:teuthology.orchestra.run.smithi195.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 736 kB/s | 3.4 MB 00:04 2024-03-25T18:31:23.767 INFO:teuthology.orchestra.run.smithi107.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 424 kB/s | 3.4 MB 00:08 2024-03-25T18:31:24.067 INFO:teuthology.orchestra.run.smithi107.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 555 kB/s | 3.1 MB 00:05 2024-03-25T18:31:24.467 INFO:teuthology.orchestra.run.smithi195.stdout:(30/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 287 kB/s | 9.7 MB 00:34 2024-03-25T18:31:26.959 INFO:teuthology.orchestra.run.smithi195.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 656 kB/s | 3.1 MB 00:04 2024-03-25T18:31:27.576 INFO:teuthology.orchestra.run.smithi195.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 538 kB/s | 20 MB 00:37 2024-03-25T18:31:27.581 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:27.582 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.5 MB/s | 58 MB 00:38 2024-03-25T18:31:27.611 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:31:27.648 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:31:27.648 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:31:28.128 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:31:28.128 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:31:29.614 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:31:29.699 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:31:29.730 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:31:29.755 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:31:29.778 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:29.795 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:29.987 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:30.042 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:30.362 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:31:30.440 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:30.532 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:30.618 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:31:30.650 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:30.713 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:30.741 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:31:30.785 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:31:30.853 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:30.877 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:30.933 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:30.968 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:31:31.074 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:31:31.119 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:31:31.159 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:31:31.192 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:31:31.220 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:31:31.253 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:31:31.570 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:31:31.592 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:31.681 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:31.713 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:31:31.757 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:31:32.213 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:31:32.244 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:32.299 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:32.366 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:31:32.397 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:31:34.519 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:34.552 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:35.115 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:35.146 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:35.222 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:35.223 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:31:35.223 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:31:35.223 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:31:35.264 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:43.747 INFO:teuthology.orchestra.run.smithi107.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 381 kB/s | 20 MB 00:53 2024-03-25T18:31:43.753 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:43.754 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.1 MB/s | 58 MB 00:53 2024-03-25T18:31:43.786 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:31:43.824 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:31:43.824 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:31:44.289 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:31:44.290 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:31:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:44.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2024-03-25T18:31:44.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2024-03-25T18:31:44.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2024-03-25T18:31:44.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2024-03-25T18:31:44.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2024-03-25T18:31:44.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /root 2024-03-25T18:31:44.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2024-03-25T18:31:44.433 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:31:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:45.387 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:45.387 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:31:45.387 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:31:45.387 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:45.387 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:45.387 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:31:45.389 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:45.466 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:45.466 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:45.768 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:31:45.846 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:31:45.885 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:31:45.910 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:31:45.941 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:45.958 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:46.107 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:46.163 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:31:46.545 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:46.615 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:46.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:31:46.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:31:46.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:31:46.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:31:46.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:31:46.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:31:46.615 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:31:46.616 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:31:46.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:31:46.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:31:46.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:31:46.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:31:46.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:31:46.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:31:46.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:31:46.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:31:46.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:46.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:31:46.617 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:31:46.645 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:46.734 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:31:46.772 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:46.835 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:46.863 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:31:46.907 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:31:46.975 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:46.998 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:47.062 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:47.097 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:31:47.233 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:31:47.273 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:31:47.314 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:31:47.346 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:31:47.375 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:31:47.407 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:31:47.674 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:47.674 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:31:47.674 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2024-03-25T18:31:47.674 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.674 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.674 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:31:47.674 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:31:47.674 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:31:47.675 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:31:47.676 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:31:47.677 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:31:47.726 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:31:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:47.844 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:47.875 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:31:47.917 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:31:48.036 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-test 2024-03-25T18:31:48.375 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:31:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:48.478 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:48.545 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:31:48.575 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:31:48.608 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:02 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:31:48.758 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:31:48.759 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:31:48.759 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-25T18:31:48.759 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-03-25T18:31:48.760 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:31:48.761 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 48 M 2024-03-25T18:31:48.761 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 204 M 2024-03-25T18:31:48.761 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:31:49.037 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T18:31:49.103 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.7 MB/s | 300 kB 00:00 2024-03-25T18:31:49.145 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T18:31:49.279 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T18:31:49.629 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 222 kB/s | 163 kB 00:00 2024-03-25T18:31:50.644 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:50.696 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:51.244 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:51.274 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:51.363 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:51.363 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:31:51.363 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:31:51.363 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:31:51.400 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:32:00.528 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:32:00.528 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-03-25T18:32:00.529 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-03-25T18:32:00.529 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-03-25T18:32:00.529 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-03-25T18:32:00.529 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-03-25T18:32:00.529 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-03-25T18:32:00.529 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-03-25T18:32:00.529 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:32:01.446 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:32:01.470 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:32:01.470 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:32:01.470 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:32:01.470 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:32:01.470 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:32:01.470 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:32:01.472 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:32:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:32:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:32:02.630 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:32:02.630 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:32:02.630 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:32:02.630 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:32:02.630 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:32:02.630 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:32:02.630 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:32:02.631 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:32:02.632 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:32:02.632 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:32:02.632 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:32:02.632 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:32:02.632 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:32:02.632 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:32:02.632 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:32:02.632 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:32:02.632 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:32:02.632 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:32:02.632 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:32:02.633 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:32:02.633 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:32:02.633 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:32:03.664 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:32:03.665 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:32:03.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:03.667 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:32:03.667 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:32:03.667 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:32:03.667 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:32:03.667 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:32:03.667 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:32:03.667 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:32:03.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2024-03-25T18:32:04.550 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:17 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:32:04.701 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:32:04.702 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:32:04.702 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-25T18:32:04.702 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:32:04.702 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:32:04.702 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:32:04.703 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-25T18:32:04.703 INFO:teuthology.orchestra.run.smithi107.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:32:04.703 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:32:04.703 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:32:04.703 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:32:04.703 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:32:04.703 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:32:04.703 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:32:04.703 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:32:04.703 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-03-25T18:32:04.703 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:32:04.704 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 48 M 2024-03-25T18:32:04.704 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 204 M 2024-03-25T18:32:04.704 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:32:04.986 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T18:32:05.035 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T18:32:05.077 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T18:32:05.219 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-25T18:32:06.494 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 97 kB/s | 163 kB 00:01 2024-03-25T18:33:30.504 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 565 kB/s | 47 MB 01:25 2024-03-25T18:33:30.506 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:30.506 INFO:teuthology.orchestra.run.smithi107.stdout:Total 575 kB/s | 48 MB 01:25 2024-03-25T18:33:30.517 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:33:30.536 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:33:30.536 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:33:30.915 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:33:30.916 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:33:31.520 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:33:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:33:31.632 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:33:31.672 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:31.712 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:33:31.740 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:36.181 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:36.254 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:38.596 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:38.596 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:33:38.596 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:33:38.596 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:38.597 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:33:38.597 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:33:39.004 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:33:39.004 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:33:39.004 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:33:39.004 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:39.004 INFO:teuthology.orchestra.run.smithi107.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:33:39.004 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:39.004 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:33:39.004 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:33:39.004 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:33:39.004 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:33:39.005 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:33:39.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2024-03-25T18:33:39.970 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:52 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:33:40.123 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:33:40.126 INFO:teuthology.orchestra.run.smithi107.stdout:========================================================================================= 2024-03-25T18:33:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-25T18:33:40.126 INFO:teuthology.orchestra.run.smithi107.stdout:========================================================================================= 2024-03-25T18:33:40.126 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:33:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:33:40.126 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:33:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:33:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout:========================================================================================= 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout:Install 36 Packages 2024-03-25T18:33:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:33:40.130 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 30 M 2024-03-25T18:33:40.130 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 107 M 2024-03-25T18:33:40.130 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:33:40.947 INFO:teuthology.orchestra.run.smithi107.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T18:33:44.297 INFO:teuthology.orchestra.run.smithi107.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 410 kB/s | 1.5 MB 00:03 2024-03-25T18:33:48.521 INFO:teuthology.orchestra.run.smithi107.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 274 kB/s | 2.1 MB 00:07 2024-03-25T18:33:49.396 INFO:teuthology.orchestra.run.smithi107.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 283 kB/s | 247 kB 00:00 2024-03-25T18:33:49.521 INFO:teuthology.orchestra.run.smithi107.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T18:33:49.563 INFO:teuthology.orchestra.run.smithi107.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:33:49.588 INFO:teuthology.orchestra.run.smithi107.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:33:49.614 INFO:teuthology.orchestra.run.smithi107.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T18:33:49.639 INFO:teuthology.orchestra.run.smithi107.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T18:33:49.681 INFO:teuthology.orchestra.run.smithi107.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T18:33:49.756 INFO:teuthology.orchestra.run.smithi107.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:33:49.789 INFO:teuthology.orchestra.run.smithi107.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:33:49.831 INFO:teuthology.orchestra.run.smithi107.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T18:33:49.873 INFO:teuthology.orchestra.run.smithi107.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:33:50.090 INFO:teuthology.orchestra.run.smithi107.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 577 kB/s | 125 kB 00:00 2024-03-25T18:33:50.366 INFO:teuthology.orchestra.run.smithi107.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 81 kB/s | 22 kB 00:00 2024-03-25T18:33:50.749 INFO:teuthology.orchestra.run.smithi107.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-03-25T18:33:50.849 INFO:teuthology.orchestra.run.smithi107.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 435 kB/s | 43 kB 00:00 2024-03-25T18:33:51.016 INFO:teuthology.orchestra.run.smithi107.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-03-25T18:33:51.083 INFO:teuthology.orchestra.run.smithi107.stdout:(20/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 473 kB/s | 4.7 MB 00:10 2024-03-25T18:33:51.183 INFO:teuthology.orchestra.run.smithi107.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-25T18:33:51.250 INFO:teuthology.orchestra.run.smithi107.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 64 kB/s | 11 kB 00:00 2024-03-25T18:33:51.277 INFO:teuthology.orchestra.run.smithi107.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-25T18:33:51.343 INFO:teuthology.orchestra.run.smithi107.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-03-25T18:33:51.368 INFO:teuthology.orchestra.run.smithi107.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-25T18:33:51.435 INFO:teuthology.orchestra.run.smithi107.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-25T18:33:51.460 INFO:teuthology.orchestra.run.smithi107.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 504 kB/s | 46 kB 00:00 2024-03-25T18:33:51.560 INFO:teuthology.orchestra.run.smithi107.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-25T18:33:51.602 INFO:teuthology.orchestra.run.smithi107.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 471 kB/s | 79 kB 00:00 2024-03-25T18:33:51.661 INFO:teuthology.orchestra.run.smithi107.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-25T18:33:51.703 INFO:teuthology.orchestra.run.smithi107.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 898 kB/s | 90 kB 00:00 2024-03-25T18:33:51.753 INFO:teuthology.orchestra.run.smithi107.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 389 kB/s | 36 kB 00:00 2024-03-25T18:33:51.853 INFO:teuthology.orchestra.run.smithi107.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-03-25T18:33:51.878 INFO:teuthology.orchestra.run.smithi107.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-03-25T18:33:51.954 INFO:teuthology.orchestra.run.smithi107.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-25T18:33:58.596 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 373 kB/s | 47 MB 02:09 2024-03-25T18:33:58.598 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:58.599 INFO:teuthology.orchestra.run.smithi195.stdout:Total 380 kB/s | 48 MB 02:09 2024-03-25T18:33:58.610 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:33:58.629 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:33:58.629 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:33:59.021 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:33:59.021 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:33:59.609 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:33:59.680 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:33:59.719 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:33:59.759 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:59.813 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:33:59.841 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:34:04.387 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:34:04.474 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:34:06.899 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:34:06.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:34:06.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:34:06.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:34:06.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:34:06.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:34:07.254 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:34:07.254 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:07.255 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:34:07.255 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:07.255 INFO:teuthology.orchestra.run.smithi195.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:34:07.255 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:07.255 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:34:07.255 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:34:07.255 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:34:07.255 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:07.255 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:34:07.592 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph 2024-03-25T18:34:08.158 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:03:22 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:34:08.318 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout:========================================================================================= 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout:========================================================================================= 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:34:08.321 INFO:teuthology.orchestra.run.smithi195.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:34:08.322 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:34:08.323 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:34:08.324 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:34:08.324 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:34:08.324 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:08.324 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:34:08.324 INFO:teuthology.orchestra.run.smithi195.stdout:========================================================================================= 2024-03-25T18:34:08.324 INFO:teuthology.orchestra.run.smithi195.stdout:Install 36 Packages 2024-03-25T18:34:08.324 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:08.324 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 30 M 2024-03-25T18:34:08.325 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 107 M 2024-03-25T18:34:08.325 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:34:09.385 INFO:teuthology.orchestra.run.smithi195.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T18:34:15.809 INFO:teuthology.orchestra.run.smithi195.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 320 kB/s | 2.1 MB 00:06 2024-03-25T18:34:16.610 INFO:teuthology.orchestra.run.smithi195.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 201 kB/s | 1.5 MB 00:07 2024-03-25T18:34:16.799 INFO:teuthology.orchestra.run.smithi107.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 529 kB/s | 17 MB 00:32 2024-03-25T18:34:16.804 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:16.804 INFO:teuthology.orchestra.run.smithi107.stdout:Total 828 kB/s | 30 MB 00:36 2024-03-25T18:34:16.829 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:34:16.871 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:34:16.871 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:34:17.302 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:34:17.302 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:34:17.576 INFO:teuthology.orchestra.run.smithi195.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 256 kB/s | 247 kB 00:00 2024-03-25T18:34:17.693 INFO:teuthology.orchestra.run.smithi195.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:34:17.735 INFO:teuthology.orchestra.run.smithi195.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:34:17.760 INFO:teuthology.orchestra.run.smithi195.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:34:17.794 INFO:teuthology.orchestra.run.smithi195.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T18:34:17.827 INFO:teuthology.orchestra.run.smithi195.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T18:34:17.870 INFO:teuthology.orchestra.run.smithi195.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T18:34:17.945 INFO:teuthology.orchestra.run.smithi195.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:34:17.978 INFO:teuthology.orchestra.run.smithi195.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:34:18.004 INFO:teuthology.orchestra.run.smithi195.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 4.9 MB/s | 124 kB 00:00 2024-03-25T18:34:18.046 INFO:teuthology.orchestra.run.smithi195.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:34:18.163 INFO:teuthology.orchestra.run.smithi195.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-25T18:34:18.430 INFO:teuthology.orchestra.run.smithi195.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 84 kB/s | 22 kB 00:00 2024-03-25T18:34:18.639 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:34:18.646 INFO:teuthology.orchestra.run.smithi195.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 698 kB/s | 151 kB 00:00 2024-03-25T18:34:18.705 INFO:teuthology.orchestra.run.smithi195.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 744 kB/s | 43 kB 00:00 2024-03-25T18:34:18.710 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:34:18.754 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:34:18.795 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:34:18.797 INFO:teuthology.orchestra.run.smithi195.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-25T18:34:18.865 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:34:18.903 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:34:18.906 INFO:teuthology.orchestra.run.smithi195.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-03-25T18:34:18.944 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:34:18.964 INFO:teuthology.orchestra.run.smithi195.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-25T18:34:19.023 INFO:teuthology.orchestra.run.smithi195.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-25T18:34:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:34:19.081 INFO:teuthology.orchestra.run.smithi195.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2024-03-25T18:34:19.114 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:34:19.132 INFO:teuthology.orchestra.run.smithi195.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 391 kB/s | 19 kB 00:00 2024-03-25T18:34:19.163 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:34:19.190 INFO:teuthology.orchestra.run.smithi195.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 338 kB/s | 20 kB 00:00 2024-03-25T18:34:19.235 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:34:19.249 INFO:teuthology.orchestra.run.smithi195.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 793 kB/s | 46 kB 00:00 2024-03-25T18:34:19.288 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:34:19.308 INFO:teuthology.orchestra.run.smithi195.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T18:34:19.327 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:34:19.375 INFO:teuthology.orchestra.run.smithi195.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-25T18:34:19.384 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:34:19.425 INFO:teuthology.orchestra.run.smithi195.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-03-25T18:34:19.440 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:34:19.492 INFO:teuthology.orchestra.run.smithi195.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-25T18:34:19.542 INFO:teuthology.orchestra.run.smithi195.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 718 kB/s | 36 kB 00:00 2024-03-25T18:34:19.609 INFO:teuthology.orchestra.run.smithi195.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-25T18:34:19.624 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:34:19.675 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:34:19.676 INFO:teuthology.orchestra.run.smithi195.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.3 MB/s | 427 kB 00:00 2024-03-25T18:34:19.705 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:34:19.727 INFO:teuthology.orchestra.run.smithi195.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-03-25T18:34:19.740 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:34:19.770 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:34:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:34:19.880 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:19.914 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:34:19.954 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:34:19.989 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:34:20.032 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:34:20.121 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:34:20.162 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:34:20.234 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:34:20.373 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:34:20.498 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:20.655 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:34:20.685 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:20.711 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:20.711 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:20.711 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:34:20.711 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:20.711 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:20.711 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:22.094 INFO:teuthology.orchestra.run.smithi195.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 377 kB/s | 4.7 MB 00:12 2024-03-25T18:34:22.135 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:22.162 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:22.162 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:22.162 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:34:22.162 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:22.162 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:22.162 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:22.595 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:22.618 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:22.618 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:22.618 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:34:22.618 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:22.618 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:22.618 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:22.829 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:22.852 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:22.852 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:22.852 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:34:22.852 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:22.853 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:22.853 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:22.917 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:24.672 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:24.672 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:34:24.672 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:34:24.672 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:34:24.672 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:34:24.672 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:34:24.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:34:24.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:34:24.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:34:24.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:34:24.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:34:24.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:34:24.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:34:24.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:34:24.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:34:24.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:34:24.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:34:24.674 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:34:24.674 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:34:24.674 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:34:24.674 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:34:24.674 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:24.674 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:34:24.674 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:34:24.674 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:34:24.674 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:34:24.674 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:34:24.675 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:34:24.675 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:34:24.675 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:34:24.675 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:24.675 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:34:24.675 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:34:24.675 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:34:24.675 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:34:24.675 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:34:25.434 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:34:25.434 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:25.434 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:34:25.434 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:25.434 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:25.434 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:25.435 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:25.435 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:25.435 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:25.435 INFO:teuthology.orchestra.run.smithi107.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:34:25.435 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:34:25.435 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:34:25.435 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:34:25.435 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:34:25.435 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:34:25.435 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:34:25.436 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:25.437 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:34:25.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2024-03-25T18:34:26.345 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:39 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:34:26.443 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:26.496 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:34:26.498 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-25T18:34:26.498 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:34:26.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2024-03-25T18:34:27.122 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:40 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:34:27.271 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:34:27.282 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:34:27.282 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-25T18:34:27.282 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:34:27.282 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:34:27.282 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:34:27.282 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:27.282 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:34:27.282 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:34:27.282 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T18:34:27.282 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:27.282 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 747 k 2024-03-25T18:34:27.283 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 754 k 2024-03-25T18:34:27.283 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:34:29.459 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 342 kB/s | 747 kB 00:02 2024-03-25T18:34:29.459 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:29.459 INFO:teuthology.orchestra.run.smithi107.stdout:Total 342 kB/s | 747 kB 00:02 2024-03-25T18:34:29.459 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:34:29.465 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:34:29.465 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:34:29.471 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:34:29.472 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:34:29.547 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:34:29.986 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:30.063 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:30.418 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:30.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:30.703 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:30.703 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:34:30.703 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:30.703 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:30.703 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:34:30.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:34:31.475 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:44 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:34:31.629 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:34:31.630 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:34:31.630 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T18:34:31.630 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:34:31.630 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:34:31.630 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:34:31.630 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:31.630 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:34:31.630 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:34:31.630 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T18:34:31.630 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:31.631 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 145 k 2024-03-25T18:34:31.631 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 436 k 2024-03-25T18:34:31.631 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:34:32.457 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 176 kB/s | 145 kB 00:00 2024-03-25T18:34:32.457 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:32.458 INFO:teuthology.orchestra.run.smithi107.stdout:Total 176 kB/s | 145 kB 00:00 2024-03-25T18:34:32.458 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:34:32.466 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:34:32.466 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:34:32.520 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:34:32.521 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:34:32.657 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:34:32.724 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:32.748 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:32.748 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:32.748 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:34:32.748 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:33.710 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:34:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:33.710 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:34:33.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2024-03-25T18:34:34.534 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:47 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:34:34.632 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:34.684 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:34:34.685 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-25T18:34:34.685 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:34:34.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:34:35.316 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:48 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:34:35.467 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:34:35.469 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2024-03-25T18:34:35.469 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-25T18:34:35.469 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2024-03-25T18:34:35.469 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:34:35.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:34:35.469 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-25T18:34:35.469 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:34:35.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:34:35.469 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:34:35.469 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout:Install 12 Packages 2024-03-25T18:34:35.470 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:35.471 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 6.9 M 2024-03-25T18:34:35.471 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 26 M 2024-03-25T18:34:35.471 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:34:36.461 INFO:teuthology.orchestra.run.smithi107.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 27 kB/s | 15 kB 00:00 2024-03-25T18:34:36.603 INFO:teuthology.orchestra.run.smithi107.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-25T18:34:36.645 INFO:teuthology.orchestra.run.smithi107.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.6 MB/s | 238 kB 00:00 2024-03-25T18:34:36.695 INFO:teuthology.orchestra.run.smithi107.stdout:(4/12): ceph-grafana-dashboards-19.0.0-1439.g5d 39 kB/s | 31 kB 00:00 2024-03-25T18:34:36.804 INFO:teuthology.orchestra.run.smithi107.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.3 MB/s | 863 kB 00:00 2024-03-25T18:34:37.221 INFO:teuthology.orchestra.run.smithi107.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 49 kB/s | 19 kB 00:00 2024-03-25T18:34:37.513 INFO:teuthology.orchestra.run.smithi107.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 671 kB/s | 548 kB 00:00 2024-03-25T18:34:37.630 INFO:teuthology.orchestra.run.smithi107.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-03-25T18:34:37.738 INFO:teuthology.orchestra.run.smithi107.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-25T18:34:37.872 INFO:teuthology.orchestra.run.smithi107.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-03-25T18:34:37.922 INFO:teuthology.orchestra.run.smithi107.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.9 MB/s | 2.0 MB 00:00 2024-03-25T18:34:41.055 INFO:teuthology.orchestra.run.smithi107.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 353 kB/s | 1.8 MB 00:05 2024-03-25T18:34:41.061 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:41.061 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.2 MB/s | 6.9 MB 00:05 2024-03-25T18:34:41.082 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:34:41.100 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:34:41.100 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:34:41.277 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:34:41.277 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:34:41.790 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:34:41.872 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:34:41.916 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:34:41.962 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:34:41.993 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:34:42.099 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:34:42.226 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:34:42.321 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:34:42.364 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:34:42.393 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:34:42.425 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:34:42.667 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:34:42.692 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:34:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:34:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:34:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:34:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:34:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:34:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:34:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:34:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:34:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:34:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:34:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:34:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:34:43.818 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:34:43.818 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:43.818 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:34:43.818 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:34:43.818 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:43.818 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:43.818 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:34:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:34:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:34:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:34:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:34:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:34:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:34:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:34:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:43.819 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:34:44.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:34:44.656 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:57 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:34:44.807 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:34:44.808 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-25T18:34:44.809 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-25T18:34:44.809 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-25T18:34:44.809 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:34:44.809 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:34:44.809 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-25T18:34:44.809 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:34:44.809 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:34:44.809 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:34:44.809 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:34:44.809 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:34:44.809 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:34:44.810 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:34:44.810 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:34:44.810 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:34:44.810 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:34:44.810 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:34:44.810 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:44.810 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:34:44.810 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-25T18:34:44.810 INFO:teuthology.orchestra.run.smithi107.stdout:Install 12 Packages 2024-03-25T18:34:44.810 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:34:44.811 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 37 M 2024-03-25T18:34:44.811 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 209 M 2024-03-25T18:34:44.811 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:34:45.162 INFO:teuthology.orchestra.run.smithi107.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 304 kB/s | 32 kB 00:00 2024-03-25T18:34:45.213 INFO:teuthology.orchestra.run.smithi107.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 654 kB/s | 17 kB 00:00 2024-03-25T18:34:45.246 INFO:teuthology.orchestra.run.smithi107.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:34:45.405 INFO:teuthology.orchestra.run.smithi107.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T18:34:45.539 INFO:teuthology.orchestra.run.smithi107.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T18:34:45.581 INFO:teuthology.orchestra.run.smithi107.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T18:34:45.639 INFO:teuthology.orchestra.run.smithi107.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-03-25T18:34:45.939 INFO:teuthology.orchestra.run.smithi107.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-25T18:34:46.465 INFO:teuthology.orchestra.run.smithi107.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 372 kB/s | 195 kB 00:00 2024-03-25T18:34:46.523 INFO:teuthology.orchestra.run.smithi107.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 911 kB/s | 803 kB 00:00 2024-03-25T18:34:46.773 INFO:teuthology.orchestra.run.smithi107.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 795 kB/s | 245 kB 00:00 2024-03-25T18:34:47.324 INFO:teuthology.orchestra.run.smithi195.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 545 kB/s | 17 MB 00:31 2024-03-25T18:34:47.328 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:47.328 INFO:teuthology.orchestra.run.smithi195.stdout:Total 779 kB/s | 30 MB 00:39 2024-03-25T18:34:47.353 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:34:47.394 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:34:47.395 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:34:47.810 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:34:47.810 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:34:49.137 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:34:49.209 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:34:49.246 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:34:49.278 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:34:49.349 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:34:49.385 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:34:49.419 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:34:49.509 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:34:49.626 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:34:49.671 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:34:49.742 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:34:49.796 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:34:49.835 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:34:49.892 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:34:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:34:50.140 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:34:50.189 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:34:50.220 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:34:50.254 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:34:50.285 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:34:50.361 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:34:50.395 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:50.428 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:34:50.468 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:34:50.503 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:34:50.546 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:34:50.635 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:34:50.677 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:34:50.757 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:34:50.894 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:34:51.018 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:51.178 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:34:51.207 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:51.233 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:51.233 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:51.233 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:34:51.233 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:51.233 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:51.233 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:52.675 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:52.701 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:52.701 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:52.701 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:34:52.701 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:52.701 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:52.701 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:53.142 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:53.165 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:53.166 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:53.166 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:34:53.166 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:53.166 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:53.166 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:53.376 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:53.399 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:53.399 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:53.400 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:34:53.400 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:53.400 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:53.400 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:53.465 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:55.154 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:55.154 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:34:55.154 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:34:55.154 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:34:55.154 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:34:55.155 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:55.156 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:34:55.157 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:34:55.157 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:34:55.157 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:34:55.157 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:34:56.133 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:34:56.133 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:34:56.134 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:34:56.135 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:56.136 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:34:56.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-base 2024-03-25T18:34:56.997 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:04:10 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:34:57.094 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:57.146 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:34:57.147 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-25T18:34:57.147 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:34:57.210 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install cephadm 2024-03-25T18:34:57.770 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:34:57.918 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:34:57.919 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:34:57.919 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-25T18:34:57.919 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:34:57.920 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:34:57.920 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:34:57.920 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:57.920 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:34:57.920 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:34:57.920 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T18:34:57.920 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:34:57.920 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 747 k 2024-03-25T18:34:57.920 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 754 k 2024-03-25T18:34:57.920 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:34:59.626 INFO:teuthology.orchestra.run.smithi195.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 438 kB/s | 747 kB 00:01 2024-03-25T18:34:59.627 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:59.627 INFO:teuthology.orchestra.run.smithi195.stdout:Total 438 kB/s | 747 kB 00:01 2024-03-25T18:34:59.627 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:34:59.634 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:34:59.634 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:34:59.640 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:34:59.640 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:34:59.719 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:35:00.188 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:35:00.274 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:35:00.496 INFO:teuthology.orchestra.run.smithi107.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 490 kB/s | 7.4 MB 00:15 2024-03-25T18:35:00.498 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:00.499 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.4 MB/s | 37 MB 00:15 2024-03-25T18:35:00.543 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:35:00.569 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:35:00.569 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:35:00.672 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:35:00.924 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:35:00.924 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:35:00.996 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:35:00.996 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:00.996 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:35:00.996 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:00.996 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:00.996 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:35:01.172 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:35:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:35:01.694 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:35:01.733 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:35:01.743 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:04:15 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:35:01.828 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:01.862 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:35:01.898 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:35:01.898 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:35:01.899 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T18:35:01.899 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:35:01.899 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:35:01.899 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:35:01.899 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:01.899 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:35:01.899 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:35:01.899 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T18:35:01.899 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:01.899 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 145 k 2024-03-25T18:35:01.900 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 436 k 2024-03-25T18:35:01.900 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:35:02.039 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:02.069 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:02.154 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:35:02.733 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 174 kB/s | 145 kB 00:00 2024-03-25T18:35:02.733 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:02.734 INFO:teuthology.orchestra.run.smithi195.stdout:Total 174 kB/s | 145 kB 00:00 2024-03-25T18:35:02.734 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:35:02.742 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:35:02.742 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:35:02.795 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:35:02.796 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:35:02.796 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:35:02.916 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:35:02.930 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:35:03.000 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:35:03.025 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:35:03.025 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:03.025 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:35:03.025 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:04.039 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:35:04.040 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:04.040 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:35:04.040 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:04.040 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:04.040 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:35:04.149 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:35:04.315 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr 2024-03-25T18:35:04.889 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:04:18 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:35:04.987 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:05.038 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:35:05.039 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-25T18:35:05.039 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:35:05.109 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:35:05.160 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:35:05.187 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:05.666 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:04:19 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:35:05.815 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:35:05.816 INFO:teuthology.orchestra.run.smithi195.stdout:====================================================================================== 2024-03-25T18:35:05.816 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-25T18:35:05.816 INFO:teuthology.orchestra.run.smithi195.stdout:====================================================================================== 2024-03-25T18:35:05.817 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:35:05.817 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:35:05.817 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-25T18:35:05.817 INFO:teuthology.orchestra.run.smithi195.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:35:05.817 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:35:05.818 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:35:05.818 INFO:teuthology.orchestra.run.smithi195.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:35:05.818 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:35:05.818 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:35:05.818 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:35:05.818 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:35:05.818 INFO:teuthology.orchestra.run.smithi195.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:35:05.818 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:35:05.818 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:35:05.819 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:05.819 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:35:05.819 INFO:teuthology.orchestra.run.smithi195.stdout:====================================================================================== 2024-03-25T18:35:05.819 INFO:teuthology.orchestra.run.smithi195.stdout:Install 12 Packages 2024-03-25T18:35:05.819 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:05.820 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 6.9 M 2024-03-25T18:35:05.820 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 26 M 2024-03-25T18:35:05.820 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:35:06.487 INFO:teuthology.orchestra.run.smithi195.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 56 kB/s | 31 kB 00:00 2024-03-25T18:35:06.728 INFO:teuthology.orchestra.run.smithi195.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 19 kB/s | 15 kB 00:00 2024-03-25T18:35:06.770 INFO:teuthology.orchestra.run.smithi195.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-03-25T18:35:06.820 INFO:teuthology.orchestra.run.smithi195.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi195.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.3 MB/s | 863 kB 00:00 2024-03-25T18:35:07.171 INFO:teuthology.orchestra.run.smithi195.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-25T18:35:07.363 INFO:teuthology.orchestra.run.smithi195.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-03-25T18:35:07.464 INFO:teuthology.orchestra.run.smithi195.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-03-25T18:35:07.572 INFO:teuthology.orchestra.run.smithi195.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 283 kB/s | 31 kB 00:00 2024-03-25T18:35:07.673 INFO:teuthology.orchestra.run.smithi195.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-25T18:35:07.683 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:07.683 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:35:07.683 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:35:07.683 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:07.683 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:35:07.683 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:07.684 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:07.684 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:35:07.684 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:35:07.684 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:35:07.684 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:35:07.684 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:35:07.714 INFO:teuthology.orchestra.run.smithi195.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.8 MB/s | 2.0 MB 00:00 2024-03-25T18:35:08.148 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:35:08.149 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:08.150 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:08.150 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:35:08.150 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:08.150 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:08.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:35:09.073 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:22 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:09.222 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:09.224 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-25T18:35:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-25T18:35:09.224 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-25T18:35:09.224 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:35:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:35:09.224 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-25T18:35:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:35:09.224 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:35:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:35:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:35:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:35:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:35:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:35:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:35:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:35:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:35:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:35:09.225 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:35:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:09.226 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:35:09.226 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-25T18:35:09.226 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2024-03-25T18:35:09.226 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:09.227 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.0 M 2024-03-25T18:35:09.227 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 26 M 2024-03-25T18:35:09.227 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:35:09.688 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 232 kB/s | 24 kB 00:00 2024-03-25T18:35:09.730 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-25T18:35:09.772 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.8 MB/s | 174 kB 00:00 2024-03-25T18:35:09.797 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T18:35:09.823 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:35:09.839 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T18:35:09.965 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2024-03-25T18:35:09.982 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 204 kB/s | 32 kB 00:00 2024-03-25T18:35:10.141 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.1 MB/s | 203 kB 00:00 2024-03-25T18:35:10.199 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-25T18:35:10.241 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-25T18:35:10.266 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-25T18:35:10.291 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 74 kB/s | 52 kB 00:00 2024-03-25T18:35:10.293 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:10.293 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-25T18:35:10.294 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:35:10.307 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:35:10.307 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:35:10.396 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:35:10.396 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:35:10.904 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:35:11.061 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:35:11.117 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:35:11.172 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:35:11.202 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:35:11.243 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:35:11.335 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:35:11.452 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:35:11.503 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:35:12.215 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:35:12.267 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:35:12.301 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:35:12.349 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:35:12.363 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:12.981 INFO:teuthology.orchestra.run.smithi195.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 259 kB/s | 1.8 MB 00:07 2024-03-25T18:35:12.985 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:12.985 INFO:teuthology.orchestra.run.smithi195.stdout:Total 980 kB/s | 6.9 MB 00:07 2024-03-25T18:35:13.005 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:35:13.024 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:35:13.024 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:35:13.124 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:13.124 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:35:13.124 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:35:13.124 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:35:13.124 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:35:13.124 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:35:13.124 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:35:13.124 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:35:13.125 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:35:13.125 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:35:13.125 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:35:13.125 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:35:13.125 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:35:13.195 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:35:13.195 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:35:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:35:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:13.471 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:35:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:35:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:35:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:35:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:35:13.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:35:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:35:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:35:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:35:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:35:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:35:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:35:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:35:13.472 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:13.472 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:13.709 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:35:13.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:35:13.789 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:35:13.833 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:35:13.947 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:35:14.052 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:35:14.149 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:35:14.277 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:14.278 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:35:14.363 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:35:14.415 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:35:14.426 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:14.427 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-25T18:35:14.427 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-25T18:35:14.427 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-25T18:35:14.427 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:35:14.427 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:35:14.428 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-25T18:35:14.428 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:35:14.428 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:35:14.428 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:35:14.428 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:35:14.428 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:35:14.428 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:14.428 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:35:14.428 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-25T18:35:14.428 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-03-25T18:35:14.428 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:14.429 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 6.8 M 2024-03-25T18:35:14.429 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 33 M 2024-03-25T18:35:14.429 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:35:14.443 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:35:14.475 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:35:14.719 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:35:14.750 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:35:14.870 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-03-25T18:35:15.095 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-03-25T18:35:15.312 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T18:35:15.337 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 238 kB/s | 58 kB 00:00 2024-03-25T18:35:15.362 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 193 kB/s | 138 kB 00:00 2024-03-25T18:35:15.388 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 677 kB/s | 51 kB 00:00 2024-03-25T18:35:15.390 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:15.390 INFO:teuthology.orchestra.run.smithi107.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2024-03-25T18:35:15.403 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:35:15.413 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:35:15.414 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:35:15.464 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:35:15.464 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:35:15.464 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:35:15.464 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:35:15.464 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:35:15.465 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:35:15.465 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:35:15.465 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:35:15.465 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:35:15.465 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:35:15.465 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:35:15.465 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:35:15.501 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:35:15.502 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:35:15.792 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:35:15.821 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:35:15.821 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:15.821 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:35:15.821 INFO:teuthology.orchestra.run.smithi195.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:35:15.821 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:15.821 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:15.822 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:15.822 INFO:teuthology.orchestra.run.smithi195.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:35:15.822 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:35:15.822 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:35:15.822 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:35:15.822 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:35:15.822 INFO:teuthology.orchestra.run.smithi195.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:35:15.822 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:35:15.822 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:35:15.822 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:15.822 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:35:15.969 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:35:16.023 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:35:16.086 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:35:16.543 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:35:16.633 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:35:16.645 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:04:30 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:35:16.695 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:35:16.718 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:16.798 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:35:16.799 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-03-25T18:35:16.799 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-25T18:35:16.799 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-03-25T18:35:16.799 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:35:16.799 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:35:16.800 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:35:16.801 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:16.801 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:35:16.801 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-03-25T18:35:16.801 INFO:teuthology.orchestra.run.smithi195.stdout:Install 12 Packages 2024-03-25T18:35:16.801 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:16.801 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 37 M 2024-03-25T18:35:16.802 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 209 M 2024-03-25T18:35:16.802 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:35:17.512 INFO:teuthology.orchestra.run.smithi195.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 205 kB/s | 32 kB 00:00 2024-03-25T18:35:17.532 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:17.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:35:17.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:35:17.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:35:17.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:35:17.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:35:17.546 INFO:teuthology.orchestra.run.smithi195.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 515 kB/s | 17 kB 00:00 2024-03-25T18:35:17.579 INFO:teuthology.orchestra.run.smithi195.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:35:17.729 INFO:teuthology.orchestra.run.smithi195.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.0 MB/s | 3.0 MB 00:00 2024-03-25T18:35:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:35:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:17.817 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:35:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:35:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:35:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:35:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:35:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:35:17.817 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:17.817 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:17.938 INFO:teuthology.orchestra.run.smithi195.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-03-25T18:35:18.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2024-03-25T18:35:18.147 INFO:teuthology.orchestra.run.smithi195.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-03-25T18:35:18.172 INFO:teuthology.orchestra.run.smithi195.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 938 kB/s | 217 kB 00:00 2024-03-25T18:35:18.619 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:31 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:18.647 INFO:teuthology.orchestra.run.smithi195.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-25T18:35:18.769 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-03-25T18:35:18.770 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:18.771 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 964 k 2024-03-25T18:35:18.771 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.8 M 2024-03-25T18:35:18.771 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:35:18.772 INFO:teuthology.orchestra.run.smithi195.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-25T18:35:18.881 INFO:teuthology.orchestra.run.smithi195.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 837 kB/s | 195 kB 00:00 2024-03-25T18:35:18.890 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 658 kB/s | 79 kB 00:00 2024-03-25T18:35:18.965 INFO:teuthology.orchestra.run.smithi195.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-25T18:35:21.590 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 314 kB/s | 885 kB 00:02 2024-03-25T18:35:21.591 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:21.592 INFO:teuthology.orchestra.run.smithi107.stdout:Total 342 kB/s | 964 kB 00:02 2024-03-25T18:35:21.592 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:35:21.615 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:35:21.615 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:35:21.691 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:35:21.691 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:35:21.875 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:35:22.016 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:35:22.081 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:22.784 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:22.785 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:35:23.081 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:35:23.081 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:23.081 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:35:23.081 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:35:23.081 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:23.081 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:23.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-volume 2024-03-25T18:35:23.928 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:36 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:24.078 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:24.079 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:24.079 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-25T18:35:24.079 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:24.079 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:35:24.080 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:35:24.080 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-25T18:35:24.080 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:35:24.080 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:24.080 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:35:24.080 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:24.080 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-03-25T18:35:24.080 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:24.080 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 332 k 2024-03-25T18:35:24.080 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.5 M 2024-03-25T18:35:24.080 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:35:24.187 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 646 kB/s | 69 kB 00:00 2024-03-25T18:35:25.253 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 225 kB/s | 264 kB 00:01 2024-03-25T18:35:25.255 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:25.255 INFO:teuthology.orchestra.run.smithi107.stdout:Total 283 kB/s | 332 kB 00:01 2024-03-25T18:35:25.255 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:35:25.265 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:35:25.265 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:35:25.310 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:35:25.310 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:35:25.463 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:35:25.644 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:35:25.681 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:25.697 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:25.697 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:25.697 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:35:25.697 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:26.360 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:35:26.683 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:35:26.683 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:26.683 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:35:26.683 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:26.683 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:35:26.684 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:26.684 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:27.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2024-03-25T18:35:27.596 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:27.746 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 127 k 2024-03-25T18:35:27.747 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 456 k 2024-03-25T18:35:27.748 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:35:28.185 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 290 kB/s | 127 kB 00:00 2024-03-25T18:35:28.185 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:28.185 INFO:teuthology.orchestra.run.smithi107.stdout:Total 289 kB/s | 127 kB 00:00 2024-03-25T18:35:28.186 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:35:28.191 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:35:28.191 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:35:28.249 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:35:28.249 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:35:28.398 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:35:28.509 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:28.917 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:29.184 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:29.184 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:29.185 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:35:29.185 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:29.185 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:29.185 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:29.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2024-03-25T18:35:30.011 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:30.113 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:30.168 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:30.169 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-25T18:35:30.169 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:30.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2024-03-25T18:35:30.802 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:30.950 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:30.950 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 31 k 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 140 k 2024-03-25T18:35:30.951 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:35:31.258 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 102 kB/s | 31 kB 00:00 2024-03-25T18:35:31.258 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:31.258 INFO:teuthology.orchestra.run.smithi107.stdout:Total 101 kB/s | 31 kB 00:00 2024-03-25T18:35:31.259 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:35:31.262 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:35:31.262 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:35:31.290 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:35:31.291 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:35:31.413 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:35:31.514 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:31.897 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:32.156 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:32.156 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:32.156 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:35:32.156 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:32.156 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:32.156 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:32.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2024-03-25T18:35:32.897 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:32.993 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:33.044 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:33.045 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-25T18:35:33.045 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:33.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2024-03-25T18:35:33.662 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:46 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:33.759 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:33.811 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:33.812 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-25T18:35:33.812 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:33.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2024-03-25T18:35:34.415 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:34.510 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:34.561 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:34.562 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-25T18:35:34.562 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:34.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:35.267 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:35.319 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:35.320 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-25T18:35:35.320 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:35.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2024-03-25T18:35:35.968 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:36.067 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:36.119 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:36.120 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-25T18:35:36.120 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:36.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2024-03-25T18:35:36.742 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:49 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:36.838 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:36.891 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:36.892 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-25T18:35:36.892 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:36.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2024-03-25T18:35:37.492 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:50 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:37.642 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:37.642 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 85 k 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 231 k 2024-03-25T18:35:37.643 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:35:38.169 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 162 kB/s | 85 kB 00:00 2024-03-25T18:35:38.170 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:38.170 INFO:teuthology.orchestra.run.smithi107.stdout:Total 161 kB/s | 85 kB 00:00 2024-03-25T18:35:38.170 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:35:38.177 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:35:38.177 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:35:38.234 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:35:38.234 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:35:38.369 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:35:38.476 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:38.887 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:39.183 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:39.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2024-03-25T18:35:40.000 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:40.153 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:40.154 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T18:35:40.154 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:40.154 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:35:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:35:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:40.155 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:35:40.155 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:40.155 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T18:35:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:40.155 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.1 M 2024-03-25T18:35:40.155 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 13 M 2024-03-25T18:35:40.155 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:35:40.512 INFO:teuthology.orchestra.run.smithi195.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 327 kB/s | 7.4 MB 00:23 2024-03-25T18:35:40.515 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:40.515 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.6 MB/s | 37 MB 00:23 2024-03-25T18:35:40.559 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:35:40.585 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:35:40.585 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:35:40.949 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:35:40.949 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:35:41.653 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:35:41.726 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:35:41.758 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:35:41.851 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:41.894 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:35:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:35:42.839 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:35:42.956 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:35:44.191 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:35:44.339 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 756 kB/s | 3.1 MB 00:04 2024-03-25T18:35:44.339 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:44.340 INFO:teuthology.orchestra.run.smithi107.stdout:Total 755 kB/s | 3.1 MB 00:04 2024-03-25T18:35:44.340 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:35:44.351 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:35:44.351 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:35:44.428 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:35:44.429 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:35:44.864 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:35:44.914 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:44.939 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:44.940 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:44.940 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:35:44.940 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:35:44.940 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:35:44.940 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:45.233 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:35:45.260 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:46.037 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:46.037 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:46.037 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:35:46.038 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:46.038 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:46.038 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:46.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2024-03-25T18:35:46.896 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:47.044 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:47.045 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:47.045 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:47.045 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:47.045 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:35:47.045 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:35:47.045 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:47.045 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:35:47.045 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:47.045 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T18:35:47.045 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:47.046 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 171 k 2024-03-25T18:35:47.046 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 494 k 2024-03-25T18:35:47.046 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:35:47.815 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:47.815 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:35:47.815 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:35:47.815 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:47.816 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:35:47.816 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:47.816 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:47.816 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:35:47.816 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:35:47.816 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:35:47.816 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:35:47.816 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:35:47.885 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 204 kB/s | 171 kB 00:00 2024-03-25T18:35:47.885 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:47.885 INFO:teuthology.orchestra.run.smithi107.stdout:Total 204 kB/s | 171 kB 00:00 2024-03-25T18:35:47.885 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:35:47.894 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:35:47.894 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:35:47.951 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:35:47.951 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:35:48.100 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:35:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:48.268 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:35:48.268 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:48.269 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:35:48.631 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:48.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:35:48.935 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:48.935 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:48.935 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:35:48.935 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:48.935 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:48.935 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:35:49.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:35:49.215 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:03 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:35:49.367 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:35:49.368 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:35:49.369 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:35:49.370 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:35:49.370 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:35:49.370 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:35:49.370 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:35:49.370 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:49.370 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:35:49.370 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-25T18:35:49.370 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2024-03-25T18:35:49.370 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:49.371 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 2.0 M 2024-03-25T18:35:49.371 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 26 M 2024-03-25T18:35:49.371 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:35:49.740 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:35:49.895 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:35:49.895 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:49.895 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T18:35:49.895 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:49.895 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:35:49.895 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:35:49.895 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-25T18:35:49.896 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:35:49.896 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:49.896 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:35:49.896 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:35:49.896 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-03-25T18:35:49.896 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:35:49.897 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 91 M 2024-03-25T18:35:49.897 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 498 M 2024-03-25T18:35:49.897 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:35:49.928 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 118 kB/s | 24 kB 00:00 2024-03-25T18:35:49.953 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 78 kB/s | 18 kB 00:00 2024-03-25T18:35:50.036 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 174 kB 00:00 2024-03-25T18:35:50.061 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-25T18:35:50.087 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:35:50.112 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-25T18:35:50.321 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 87 kB/s | 52 kB 00:00 2024-03-25T18:35:50.346 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 124 kB/s | 32 kB 00:00 2024-03-25T18:35:50.371 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-25T18:35:50.538 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-25T18:35:50.697 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-03-25T18:35:50.730 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2024-03-25T18:35:50.764 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): python3-google-auth-2.28.2-1.el9.noarc 458 kB/s | 203 kB 00:00 2024-03-25T18:35:50.766 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:50.766 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-25T18:35:50.767 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:35:50.781 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:35:50.781 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:35:50.879 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:35:50.879 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:35:51.400 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:35:51.579 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:35:51.640 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:35:51.704 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:35:51.741 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:35:51.782 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:35:51.877 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:35:51.985 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:35:52.042 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:35:52.768 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:35:52.839 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:35:52.872 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:35:52.930 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:35:52.943 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:53.743 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:53.743 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:35:53.743 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:35:53.743 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:35:53.743 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:35:53.744 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:35:53.744 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:35:53.744 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:35:53.744 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:35:53.744 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:35:53.744 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:35:53.744 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:35:53.744 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:35:54.097 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:35:54.098 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:54.098 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:35:54.098 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:54.098 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:35:54.098 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:35:54.098 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:35:54.098 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:35:54.098 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:35:54.098 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:35:54.098 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:35:54.098 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:35:54.099 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:35:54.099 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:35:54.099 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:35:54.099 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:35:54.099 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:54.099 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:35:54.336 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:35:54.901 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:08 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:35:55.058 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:35:55.059 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-03-25T18:35:55.060 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:55.061 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 6.8 M 2024-03-25T18:35:55.061 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 33 M 2024-03-25T18:35:55.061 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:35:55.882 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 667 kB/s | 248 kB 00:00 2024-03-25T18:35:56.032 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-25T18:35:56.141 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-03-25T18:35:56.166 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 432 kB/s | 58 kB 00:00 2024-03-25T18:35:56.191 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 202 kB/s | 138 kB 00:00 2024-03-25T18:35:56.217 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 674 kB/s | 51 kB 00:00 2024-03-25T18:35:56.219 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:56.219 INFO:teuthology.orchestra.run.smithi195.stdout:Total 5.9 MB/s | 6.8 MB 00:01 2024-03-25T18:35:56.232 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:35:56.242 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:35:56.242 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:35:56.333 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:35:56.341 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:35:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:35:56.809 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:35:56.860 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:35:57.374 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:35:57.446 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:35:57.508 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:35:57.539 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:35:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:35:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:35:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:35:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:35:58.639 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:35:58.640 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:58.640 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:35:58.640 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:58.640 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:35:58.640 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:35:58.640 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:35:58.640 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:35:58.640 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:35:58.640 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:58.640 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:35:58.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-fuse 2024-03-25T18:35:59.449 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:35:59.599 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:35:59.599 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:35:59.599 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-25T18:35:59.599 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:35:59.599 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:35:59.599 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:35:59.600 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-25T18:35:59.600 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:35:59.600 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:59.600 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:35:59.600 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:35:59.600 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-03-25T18:35:59.600 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:35:59.600 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 964 k 2024-03-25T18:35:59.600 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 2.8 M 2024-03-25T18:35:59.601 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:35:59.743 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 551 kB/s | 79 kB 00:00 2024-03-25T18:36:01.360 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 503 kB/s | 885 kB 00:01 2024-03-25T18:36:01.362 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:01.362 INFO:teuthology.orchestra.run.smithi195.stdout:Total 547 kB/s | 964 kB 00:01 2024-03-25T18:36:01.362 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:36:01.386 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:36:01.386 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:36:01.462 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:36:01.462 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:36:01.640 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:36:01.778 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:36:01.851 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:36:02.508 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:36:02.508 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:36:02.814 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:36:02.814 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:02.814 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:36:02.814 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:36:02.814 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:02.814 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:03.117 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-volume 2024-03-25T18:36:03.693 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:17 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:03.844 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:03.844 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-25T18:36:03.845 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:03.845 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:36:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:36:03.845 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-25T18:36:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:36:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:03.845 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:36:03.845 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:03.845 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-03-25T18:36:03.845 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:03.846 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 332 k 2024-03-25T18:36:03.846 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 1.5 M 2024-03-25T18:36:03.846 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:36:04.556 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 97 kB/s | 69 kB 00:00 2024-03-25T18:36:04.965 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 236 kB/s | 264 kB 00:01 2024-03-25T18:36:04.966 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:04.967 INFO:teuthology.orchestra.run.smithi195.stdout:Total 296 kB/s | 332 kB 00:01 2024-03-25T18:36:04.967 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:36:04.976 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:36:04.976 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:36:05.019 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:36:05.020 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:36:05.169 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:36:05.328 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:36:05.361 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:36:05.376 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:36:05.376 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:36:05.376 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:36:05.376 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:06.099 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:36:06.454 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:36:06.454 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:06.454 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:36:06.454 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:36:06.454 INFO:teuthology.orchestra.run.smithi195.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:36:06.454 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:06.454 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:06.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados-devel 2024-03-25T18:36:07.354 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:07.506 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:07.507 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:07.507 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T18:36:07.507 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:07.507 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:36:07.507 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:36:07.507 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:07.507 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:36:07.507 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:07.507 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T18:36:07.507 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:07.507 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 127 k 2024-03-25T18:36:07.508 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 456 k 2024-03-25T18:36:07.508 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:36:08.348 INFO:teuthology.orchestra.run.smithi195.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 151 kB/s | 127 kB 00:00 2024-03-25T18:36:08.348 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:08.349 INFO:teuthology.orchestra.run.smithi195.stdout:Total 150 kB/s | 127 kB 00:00 2024-03-25T18:36:08.349 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:36:08.355 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:36:08.355 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:36:08.413 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:36:08.413 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:36:08.566 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:36:08.656 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:09.109 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:09.421 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:09.421 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:09.421 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:36:09.421 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:09.421 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:09.421 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:09.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs2 2024-03-25T18:36:10.218 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:10.316 INFO:teuthology.orchestra.run.smithi195.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:10.369 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:10.370 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-25T18:36:10.371 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:10.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs-devel 2024-03-25T18:36:10.998 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:11.148 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:11.149 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:11.149 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T18:36:11.149 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:11.149 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:36:11.149 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:36:11.149 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:11.149 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:36:11.149 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:11.149 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T18:36:11.149 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:11.150 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 31 k 2024-03-25T18:36:11.150 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 140 k 2024-03-25T18:36:11.150 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:36:11.601 INFO:teuthology.orchestra.run.smithi195.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 69 kB/s | 31 kB 00:00 2024-03-25T18:36:11.602 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:11.602 INFO:teuthology.orchestra.run.smithi195.stdout:Total 69 kB/s | 31 kB 00:00 2024-03-25T18:36:11.602 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:36:11.605 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:36:11.605 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:36:11.634 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:36:11.634 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:36:11.748 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:36:11.849 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:12.216 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:12.503 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:12.503 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:12.503 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:36:12.503 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:12.503 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:12.503 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:12.659 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados2 2024-03-25T18:36:13.225 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:27 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:13.322 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:13.375 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:13.376 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-25T18:36:13.376 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:13.424 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librbd1 2024-03-25T18:36:13.985 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:27 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:14.082 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:14.133 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:14.134 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-25T18:36:14.134 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:14.181 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rados 2024-03-25T18:36:14.740 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:28 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:14.836 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:14.888 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:14.889 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-25T18:36:14.889 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:14.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rgw 2024-03-25T18:36:15.532 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:29 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:15.633 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:15.687 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:15.688 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-25T18:36:15.688 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:15.744 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-cephfs 2024-03-25T18:36:16.325 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:30 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:16.425 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:16.477 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:16.478 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-25T18:36:16.478 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:16.520 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rbd 2024-03-25T18:36:17.085 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:31 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:17.181 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:17.233 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:17.234 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-25T18:36:17.234 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:17.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-fuse 2024-03-25T18:36:17.852 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:31 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:18.006 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:18.006 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:18.007 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-03-25T18:36:18.007 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:18.007 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:36:18.007 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:36:18.007 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:18.007 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:36:18.007 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:18.007 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T18:36:18.007 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:18.008 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 85 k 2024-03-25T18:36:18.008 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 231 k 2024-03-25T18:36:18.008 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:36:19.096 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 78 kB/s | 85 kB 00:01 2024-03-25T18:36:19.096 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:19.097 INFO:teuthology.orchestra.run.smithi195.stdout:Total 78 kB/s | 85 kB 00:01 2024-03-25T18:36:19.097 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:36:19.105 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:36:19.105 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:36:19.164 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:36:19.165 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:36:19.304 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:36:19.395 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:19.819 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:20.119 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:20.119 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:20.119 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:36:20.119 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:20.119 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:20.119 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:20.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-mirror 2024-03-25T18:36:20.926 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:34 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:21.078 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:21.079 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T18:36:21.079 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:21.079 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:36:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:36:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:21.080 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:36:21.080 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:21.080 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T18:36:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:21.080 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.1 M 2024-03-25T18:36:21.080 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 13 M 2024-03-25T18:36:21.080 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:36:30.565 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 565 kB/s | 22 MB 00:40 2024-03-25T18:36:30.923 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 321 kB/s | 3.1 MB 00:09 2024-03-25T18:36:30.924 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:30.924 INFO:teuthology.orchestra.run.smithi195.stdout:Total 321 kB/s | 3.1 MB 00:09 2024-03-25T18:36:30.925 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:36:30.935 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:36:30.936 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:36:31.012 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:36:31.013 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:36:31.440 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:36:31.507 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:31.532 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:31.532 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:36:31.532 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:36:31.532 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:36:31.532 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:36:31.532 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:32.685 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:32.685 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:32.685 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:36:32.685 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:32.685 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:32.685 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:32.920 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-nbd 2024-03-25T18:36:33.500 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:47 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:33.649 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:33.650 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-03-25T18:36:33.650 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:33.650 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:36:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:36:33.650 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:33.650 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:36:33.651 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:33.651 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T18:36:33.651 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:33.651 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 171 k 2024-03-25T18:36:33.651 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 494 k 2024-03-25T18:36:33.651 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:36:34.352 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 245 kB/s | 171 kB 00:00 2024-03-25T18:36:34.353 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:34.353 INFO:teuthology.orchestra.run.smithi195.stdout:Total 244 kB/s | 171 kB 00:00 2024-03-25T18:36:34.353 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:36:34.361 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:36:34.361 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:36:34.418 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:36:34.418 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:36:34.565 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:36:34.659 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:35.051 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:35.349 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:35.349 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:35.350 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:36:35.350 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:35.350 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:35.350 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:36:35.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:36:36.186 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:05:50 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:36:36.339 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:36:36.340 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:36.340 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T18:36:36.340 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:36.340 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:36:36.340 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:36:36.340 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-25T18:36:36.340 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:36:36.340 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:36.340 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:36:36.340 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:36:36.341 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-03-25T18:36:36.341 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:36:36.341 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 91 M 2024-03-25T18:36:36.341 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 498 M 2024-03-25T18:36:36.341 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:37:34.249 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 397 kB/s | 22 MB 00:57 2024-03-25T18:38:04.521 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 518 kB/s | 68 MB 02:14 2024-03-25T18:38:04.522 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:38:04.522 INFO:teuthology.orchestra.run.smithi107.stdout:Total 689 kB/s | 91 MB 02:14 2024-03-25T18:38:04.522 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:38:04.526 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:38:04.526 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:38:05.104 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:38:05.104 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:38:08.213 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:38:15.386 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:38:15.529 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:38:18.965 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:38:18.966 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:38:19.229 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:38:19.230 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:38:19.230 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:38:19.230 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:19.230 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:19.230 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:38:19.230 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:38:19.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:38:20.021 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:07:33 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:38:20.169 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:38:20.170 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:38:20.170 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T18:38:20.170 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:38:20.170 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:38:20.170 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:38:20.170 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:38:20.170 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:38:20.170 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:38:20.171 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T18:38:20.171 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:38:20.171 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 416 M 2024-03-25T18:38:20.171 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.7 G 2024-03-25T18:38:20.171 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:39:12.544 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 447 kB/s | 68 MB 02:36 2024-03-25T18:39:12.545 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:39:12.545 INFO:teuthology.orchestra.run.smithi195.stdout:Total 594 kB/s | 91 MB 02:36 2024-03-25T18:39:12.545 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:39:12.549 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:39:12.550 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:39:13.129 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:39:13.129 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:39:16.214 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:39:23.281 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:39:23.368 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:39:26.678 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:39:26.679 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:39:26.948 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:39:26.948 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:39:26.949 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:39:26.949 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:26.949 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:26.949 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:39:26.949 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:39:27.169 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:39:27.738 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:08:41 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:39:27.894 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:39:27.895 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:39:27.895 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T18:39:27.895 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:39:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:39:27.896 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:39:27.896 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:39:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:39:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:39:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T18:39:27.896 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:39:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 416 M 2024-03-25T18:39:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 1.7 G 2024-03-25T18:39:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:51:06.293 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 556 kB/s | 416 MB 12:46 2024-03-25T18:51:06.293 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:06.293 INFO:teuthology.orchestra.run.smithi107.stdout:Total 556 kB/s | 416 MB 12:46 2024-03-25T18:51:06.293 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:51:06.297 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:51:06.297 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:51:08.780 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:51:08.781 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:51:51.473 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:51:51.751 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:52:00.063 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:52:00.312 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:52:00.312 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:52:00.312 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:52:00.312 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:52:00.312 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:52:00.312 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:52:00.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:52:01.118 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:21:14 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:52:01.278 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:52:01.279 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-25T18:52:01.279 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T18:52:01.279 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-25T18:52:01.279 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:52:01.279 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:52:01.280 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:52:01.280 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:52:01.280 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-25T18:52:01.280 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T18:52:01.280 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:52:01.280 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.4 M 2024-03-25T18:52:01.280 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 6.4 M 2024-03-25T18:52:01.280 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:52:06.120 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 302 kB/s | 1.4 MB 00:04 2024-03-25T18:52:06.120 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:06.121 INFO:teuthology.orchestra.run.smithi107.stdout:Total 302 kB/s | 1.4 MB 00:04 2024-03-25T18:52:06.121 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:52:06.125 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:52:06.125 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:52:06.172 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:52:06.172 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:52:06.428 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:52:06.518 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:52:06.914 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:52:07.191 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:52:07.191 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:52:07.191 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:52:07.191 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:52:07.191 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:52:07.191 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:52:07.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:52:07.958 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:21:20 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:52:08.105 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:52:08.106 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:52:08.106 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T18:52:08.106 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:52:08.106 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:52:08.106 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:52:08.106 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:52:08.106 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:52:08.106 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:52:08.106 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T18:52:08.106 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:52:08.107 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 176 M 2024-03-25T18:52:08.107 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 753 M 2024-03-25T18:52:08.107 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T18:55:54.210 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 432 kB/s | 416 MB 16:26 2024-03-25T18:55:54.211 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:55:54.211 INFO:teuthology.orchestra.run.smithi195.stdout:Total 432 kB/s | 416 MB 16:26 2024-03-25T18:55:54.211 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:55:54.214 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:55:54.214 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:55:56.686 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:55:56.686 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:56:38.896 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:56:38.988 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:56:47.015 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:56:47.286 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:56:47.286 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:56:47.286 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:56:47.286 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:56:47.286 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:56:47.286 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:56:47.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:56:48.119 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:26:02 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:56:48.278 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:56:48.279 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-25T18:56:48.279 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T18:56:48.279 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-25T18:56:48.279 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:56:48.279 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:56:48.280 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:56:48.280 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:56:48.280 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-25T18:56:48.280 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T18:56:48.280 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:56:48.280 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 1.4 M 2024-03-25T18:56:48.280 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 6.4 M 2024-03-25T18:56:48.280 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:56:52.732 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 328 kB/s | 1.4 MB 00:04 2024-03-25T18:56:52.733 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:56:52.733 INFO:teuthology.orchestra.run.smithi195.stdout:Total 328 kB/s | 1.4 MB 00:04 2024-03-25T18:56:52.734 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T18:56:52.738 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T18:56:52.738 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T18:56:52.784 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T18:56:52.785 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T18:56:53.046 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T18:56:53.155 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:56:53.585 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:56:53.868 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:56:53.869 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:56:53.869 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T18:56:53.869 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:56:53.869 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:56:53.869 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T18:56:54.072 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:56:54.632 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:26:08 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T18:56:54.780 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 176 M 2024-03-25T18:56:54.781 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 753 M 2024-03-25T18:56:54.782 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T18:58:19.184 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 485 kB/s | 176 MB 06:11 2024-03-25T18:58:19.184 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:58:19.184 INFO:teuthology.orchestra.run.smithi107.stdout:Total 485 kB/s | 176 MB 06:11 2024-03-25T18:58:19.184 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T18:58:19.187 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T18:58:19.187 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T18:58:20.244 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T18:58:20.244 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T18:58:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T18:58:38.370 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:58:43.060 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:58:43.325 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:58:43.326 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:58:43.326 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T18:58:43.326 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:58:43.326 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:58:43.326 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T18:58:43.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:58:44.111 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:27:57 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T18:58:44.260 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T18:58:44.260 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:58:44.260 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T18:58:44.260 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:58:44.260 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T18:58:44.260 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:58:44.260 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:58:44.260 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T18:58:44.260 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T18:58:44.261 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T18:58:44.261 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T18:58:44.261 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 672 M 2024-03-25T18:58:44.261 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 3.1 G 2024-03-25T18:58:44.261 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:02:24.285 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 546 kB/s | 176 MB 05:29 2024-03-25T19:02:24.286 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:02:24.286 INFO:teuthology.orchestra.run.smithi195.stdout:Total 546 kB/s | 176 MB 05:29 2024-03-25T19:02:24.286 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:02:24.289 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:02:24.289 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:02:25.348 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:02:25.348 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:02:43.297 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:02:43.397 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:02:48.101 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:02:48.373 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:02:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:02:48.374 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:02:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:02:48.374 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:02:48.374 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:02:48.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:02:49.156 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:32:03 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:02:49.305 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:02:49.305 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 672 M 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 3.1 G 2024-03-25T19:02:49.306 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:20:44.715 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 521 kB/s | 672 MB 22:00 2024-03-25T19:20:44.715 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:20:44.716 INFO:teuthology.orchestra.run.smithi107.stdout:Total 521 kB/s | 672 MB 22:00 2024-03-25T19:20:44.716 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:20:44.720 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:20:44.720 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:20:48.671 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:20:48.671 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:21:56.333 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 600 kB/s | 672 MB 19:07 2024-03-25T19:21:56.333 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:56.333 INFO:teuthology.orchestra.run.smithi195.stdout:Total 600 kB/s | 672 MB 19:07 2024-03-25T19:21:56.333 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:21:56.337 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:21:56.338 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:21:59.184 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:21:59.293 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:00.314 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:22:00.314 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:22:08.884 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:22:09.147 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:22:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:09.147 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:22:09.147 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:22:09.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:22:09.997 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:51:22 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:22:10.093 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:22:10.146 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:22:10.147 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-25T19:22:10.147 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:22:10.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:22:10.775 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:51:23 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:22:10.923 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:22:10.923 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:22:10.923 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:22:10.923 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:22:10.923 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:22:10.923 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:22:10.923 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:22:10.923 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:22:10.923 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:22:10.923 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:22:10.924 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:22:10.924 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 20 M 2024-03-25T19:22:10.924 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 103 M 2024-03-25T19:22:10.924 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:22:50.631 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 508 kB/s | 20 MB 00:39 2024-03-25T19:22:50.632 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:50.632 INFO:teuthology.orchestra.run.smithi107.stdout:Total 508 kB/s | 20 MB 00:39 2024-03-25T19:22:50.632 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:22:50.636 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:22:50.636 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:22:50.785 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:22:50.785 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:22:53.004 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:22:53.179 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:54.151 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:54.412 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:54.412 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:22:54.412 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:22:54.412 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:54.412 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:22:54.412 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:22:54.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:22:55.162 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:52:08 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:22:55.319 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:22:55.320 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:22:55.320 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:22:55.320 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:22:55.320 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:22:55.320 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:22:55.320 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:22:55.320 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:22:55.320 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:22:55.320 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:22:55.320 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:22:55.321 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 48 M 2024-03-25T19:22:55.321 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 173 M 2024-03-25T19:22:55.321 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:23:11.334 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:23:11.515 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:20.684 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:23:20.956 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:23:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:20.956 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:23:20.956 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:23:21.239 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:23:21.805 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:52:35 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:23:21.901 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:23:21.953 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:23:21.954 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-25T19:23:21.954 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:23:22.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:23:22.594 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:52:36 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:23:22.742 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:23:22.743 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:23:22.743 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:23:22.743 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:23:22.743 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:23:22.743 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:23:22.743 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:23:22.743 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:23:22.743 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:23:22.743 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:23:22.743 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:23:22.744 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 20 M 2024-03-25T19:23:22.744 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 103 M 2024-03-25T19:23:22.744 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:23:55.057 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 624 kB/s | 20 MB 00:32 2024-03-25T19:23:55.057 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:55.058 INFO:teuthology.orchestra.run.smithi195.stdout:Total 624 kB/s | 20 MB 00:32 2024-03-25T19:23:55.058 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:23:55.062 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:23:55.062 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:23:55.213 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:23:55.213 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:23:57.447 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:23:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:58.772 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:59.042 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:59.042 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:23:59.042 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:23:59.042 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:59.042 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:23:59.042 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:23:59.227 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:23:59.791 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:53:13 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:23:59.940 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:23:59.941 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:23:59.941 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:23:59.942 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:23:59.942 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:23:59.942 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:23:59.942 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:23:59.942 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:23:59.942 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:23:59.942 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:23:59.942 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:23:59.942 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 48 M 2024-03-25T19:23:59.942 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 173 M 2024-03-25T19:23:59.942 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:24:40.872 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 463 kB/s | 48 MB 01:45 2024-03-25T19:24:40.873 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:40.873 INFO:teuthology.orchestra.run.smithi107.stdout:Total 463 kB/s | 48 MB 01:45 2024-03-25T19:24:40.873 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:24:40.876 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:24:40.876 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:24:41.191 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:24:41.191 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:24:45.918 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:24:46.071 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:47.389 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:24:47.656 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:24:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:47.656 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:24:47.656 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:24:47.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:24:48.391 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:54:01 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:24:48.540 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:24:48.540 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:24:48.541 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:24:48.541 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:24:48.541 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:24:48.541 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:24:48.541 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:24:48.541 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:24:48.541 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:24:48.541 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:24:48.541 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:24:48.542 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 99 M 2024-03-25T19:24:48.542 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 449 M 2024-03-25T19:24:48.542 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:25:27.972 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 555 kB/s | 48 MB 01:28 2024-03-25T19:25:27.973 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:27.973 INFO:teuthology.orchestra.run.smithi195.stdout:Total 555 kB/s | 48 MB 01:28 2024-03-25T19:25:27.974 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:25:27.977 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:25:27.977 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:25:28.293 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:25:28.293 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:25:33.110 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:25:33.263 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:34.637 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:34.921 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:34.921 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:25:34.921 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:25:34.921 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:34.922 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:25:34.922 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:25:35.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:25:35.690 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:54:49 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:25:35.839 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 99 M 2024-03-25T19:25:35.840 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 449 M 2024-03-25T19:25:35.841 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:26:02.543 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 99 MB 01:14 2024-03-25T19:26:02.544 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:02.544 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.3 MB/s | 99 MB 01:14 2024-03-25T19:26:02.544 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:26:02.547 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:26:02.547 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:26:03.157 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:26:03.157 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:26:13.414 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:26:13.514 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:26:16.413 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:26:16.676 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:26:16.676 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:26:16.676 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:26:16.677 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:16.677 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:26:16.677 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:26:16.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:26:17.436 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:55:30 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:26:17.585 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:26:17.586 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:26:17.586 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:26:17.586 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:26:17.586 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:26:17.586 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:26:17.587 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:26:17.587 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:26:17.587 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:26:17.587 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:26:17.587 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:26:17.587 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 287 M 2024-03-25T19:26:17.587 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.2 G 2024-03-25T19:26:17.587 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:27:20.679 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 962 kB/s | 99 MB 01:44 2024-03-25T19:27:20.679 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:20.680 INFO:teuthology.orchestra.run.smithi195.stdout:Total 962 kB/s | 99 MB 01:44 2024-03-25T19:27:20.680 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:27:20.683 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:27:20.683 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:27:21.294 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:27:21.294 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:27:31.558 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:27:31.651 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:34.512 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:34.802 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:34.802 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:27:34.802 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:27:34.803 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:34.803 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:27:34.803 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:27:35.000 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:27:35.566 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:56:49 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:27:35.715 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:27:35.716 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:27:35.716 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:27:35.716 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:27:35.716 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:27:35.716 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:27:35.716 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:27:35.716 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:27:35.716 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:27:35.717 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:27:35.717 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:27:35.717 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 287 M 2024-03-25T19:27:35.717 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 1.2 G 2024-03-25T19:27:35.717 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:30:43.209 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.1 MB/s | 287 MB 04:25 2024-03-25T19:30:43.210 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:43.210 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.1 MB/s | 287 MB 04:25 2024-03-25T19:30:43.211 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:30:43.214 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:30:43.215 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:30:44.927 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:30:44.928 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:31:14.348 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:31:14.446 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:21.632 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:21.932 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:21.932 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:21.932 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:31:21.932 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:21.932 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:21.933 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:31:22.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:31:22.706 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:00:35 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:31:22.853 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 5.9 M 2024-03-25T19:31:22.854 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 28 M 2024-03-25T19:31:22.855 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:31:29.763 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 881 kB/s | 5.9 MB 00:06 2024-03-25T19:31:29.764 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:29.765 INFO:teuthology.orchestra.run.smithi107.stdout:Total 881 kB/s | 5.9 MB 00:06 2024-03-25T19:31:29.765 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:31:29.769 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:31:29.769 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:31:29.840 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:31:29.840 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:31:30.572 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:31:30.678 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:31.283 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:31.547 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:31.547 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:31.547 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:31:31.547 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:31.547 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:31.547 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:31:31.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:31:32.299 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:00:45 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:31:32.449 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 109 k 2024-03-25T19:31:32.450 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 440 k 2024-03-25T19:31:32.451 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.2 MB/s | 287 MB 03:57 2024-03-25T19:31:32.767 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.2 MB/s | 287 MB 03:57 2024-03-25T19:31:32.768 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:31:32.772 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:31:33.158 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 154 kB/s | 109 kB 00:00 2024-03-25T19:31:33.158 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:33.159 INFO:teuthology.orchestra.run.smithi107.stdout:Total 153 kB/s | 109 kB 00:00 2024-03-25T19:31:33.159 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:31:33.162 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:31:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:31:33.200 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:31:33.200 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:31:33.322 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:31:33.431 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:34.049 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:34.049 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:34.050 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:31:34.050 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:34.050 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:34.050 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:31:34.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:31:34.479 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:31:34.479 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:31:34.791 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:00:47 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:31:34.939 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:31:34.940 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:34.940 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:31:34.940 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:34.940 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:31:34.941 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:31:34.941 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:34.941 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:31:34.941 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:34.941 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:31:34.941 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:34.941 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 5.4 M 2024-03-25T19:31:34.941 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 25 M 2024-03-25T19:31:34.941 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:31:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 602 kB/s | 5.4 MB 00:09 2024-03-25T19:31:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:Total 602 kB/s | 5.4 MB 00:09 2024-03-25T19:31:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:31:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:31:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:31:44.116 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:31:44.116 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:31:44.780 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:31:44.874 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:31:45.456 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:31:45.714 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:31:45.714 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:45.714 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:31:45.714 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:45.714 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:45.714 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:31:45.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2-debuginfo 2024-03-25T19:31:46.466 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:00:59 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:31:46.614 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 44 M 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 253 M 2024-03-25T19:31:46.615 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:32:03.829 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:32:03.929 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:11.231 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:11.528 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:32:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:11.528 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:11.529 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:32:11.760 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:32:12.326 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:01:26 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:32:12.475 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:32:12.476 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:32:12.476 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:12.476 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:32:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:32:12.476 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:12.476 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:32:12.476 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:12.476 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:32:12.477 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:12.477 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 5.9 M 2024-03-25T19:32:12.477 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 28 M 2024-03-25T19:32:12.477 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:32:21.232 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 695 kB/s | 5.9 MB 00:08 2024-03-25T19:32:21.233 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:21.234 INFO:teuthology.orchestra.run.smithi195.stdout:Total 695 kB/s | 5.9 MB 00:08 2024-03-25T19:32:21.234 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:32:21.238 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:32:21.238 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:32:21.309 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:32:21.309 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:32:22.042 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:32:22.139 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:22.732 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:23.016 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:23.016 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:23.016 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:32:23.016 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:23.016 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:23.016 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:32:23.206 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:32:23.770 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:01:37 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:32:23.921 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:32:23.922 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:23.922 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:32:23.922 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:23.922 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:32:23.922 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:32:23.922 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:23.923 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:32:23.923 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:23.923 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:32:23.923 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:23.923 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 109 k 2024-03-25T19:32:23.923 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 440 k 2024-03-25T19:32:23.923 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:32:24.527 INFO:teuthology.orchestra.run.smithi195.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 180 kB/s | 109 kB 00:00 2024-03-25T19:32:24.527 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:24.527 INFO:teuthology.orchestra.run.smithi195.stdout:Total 180 kB/s | 109 kB 00:00 2024-03-25T19:32:24.527 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:32:24.531 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:32:24.531 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:32:24.565 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:32:24.565 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:32:24.681 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:32:24.767 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:32:25.167 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:32:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:32:25.440 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:25.440 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:32:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:25.441 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:25.441 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:32:25.627 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:32:26.190 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:01:40 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:32:26.339 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:32:26.340 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:26.340 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:32:26.340 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:26.340 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:32:26.340 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:32:26.340 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:26.340 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:32:26.340 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:26.340 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:32:26.340 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:26.341 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 5.4 M 2024-03-25T19:32:26.341 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 25 M 2024-03-25T19:32:26.341 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:32:35.428 INFO:teuthology.orchestra.run.smithi195.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 603 kB/s | 5.4 MB 00:09 2024-03-25T19:32:35.430 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:35.430 INFO:teuthology.orchestra.run.smithi195.stdout:Total 603 kB/s | 5.4 MB 00:09 2024-03-25T19:32:35.430 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:32:35.434 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:32:35.434 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:32:35.502 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:32:35.503 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:32:36.193 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:32:36.286 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:32:36.827 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:32:37.105 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:32:37.105 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:37.105 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:32:37.105 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:37.105 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:37.105 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:32:37.301 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados2-debuginfo 2024-03-25T19:32:37.862 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:01:51 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:32:38.012 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:32:38.012 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:38.012 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:32:38.013 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:38.013 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:32:38.013 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:32:38.013 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:38.013 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:32:38.013 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:32:38.013 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:32:38.013 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:32:38.013 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 44 M 2024-03-25T19:32:38.013 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 253 M 2024-03-25T19:32:38.013 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:32:38.144 INFO:teuthology.orchestra.run.smithi107.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 880 kB/s | 44 MB 00:51 2024-03-25T19:32:38.145 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:38.145 INFO:teuthology.orchestra.run.smithi107.stdout:Total 880 kB/s | 44 MB 00:51 2024-03-25T19:32:38.145 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:32:38.149 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:32:38.149 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:32:38.450 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:32:38.450 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:32:43.491 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:32:43.593 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:45.384 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:45.642 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:45.642 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:32:45.642 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:32:45.642 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:45.642 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:32:45.642 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:32:45.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:32:46.397 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:01:59 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:32:46.549 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:32:46.549 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:32:46.550 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:32:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:32:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:32:46.550 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:32:46.550 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:32:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:32:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:32:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:32:46.550 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:32:46.551 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 52 M 2024-03-25T19:32:46.551 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 308 M 2024-03-25T19:32:46.551 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:33:09.553 INFO:teuthology.orchestra.run.smithi195.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.4 MB/s | 44 MB 00:31 2024-03-25T19:33:09.554 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:09.554 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.4 MB/s | 44 MB 00:31 2024-03-25T19:33:09.554 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:33:09.558 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:33:09.558 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:33:09.860 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:33:09.861 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:33:14.935 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:33:15.026 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:33:16.851 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:33:17.132 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:33:17.132 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:17.132 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:33:17.132 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:17.132 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:17.133 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:33:17.345 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:33:17.913 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:02:31 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:33:18.061 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:33:18.062 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:18.062 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:33:18.062 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:18.062 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:33:18.063 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:33:18.063 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:18.063 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:33:18.063 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:18.063 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:33:18.063 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:18.063 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 52 M 2024-03-25T19:33:18.063 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 308 M 2024-03-25T19:33:18.063 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:33:27.447 INFO:teuthology.orchestra.run.smithi107.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.3 MB/s | 52 MB 00:40 2024-03-25T19:33:27.448 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:27.448 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.3 MB/s | 52 MB 00:40 2024-03-25T19:33:27.448 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:33:27.452 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:33:27.452 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:33:27.801 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:33:27.801 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:33:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:33:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:36.060 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:36.324 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:33:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:36.324 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:33:36.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:33:37.085 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:02:50 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:33:37.235 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:33:37.235 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:37.235 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:33:37.235 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:37.235 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:33:37.235 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:33:37.236 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:37.236 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:33:37.236 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:37.236 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:33:37.236 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:37.236 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 615 k 2024-03-25T19:33:37.236 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.1 M 2024-03-25T19:33:37.236 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:33:38.741 INFO:teuthology.orchestra.run.smithi107.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 409 kB/s | 615 kB 00:01 2024-03-25T19:33:38.742 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:38.742 INFO:teuthology.orchestra.run.smithi107.stdout:Total 408 kB/s | 615 kB 00:01 2024-03-25T19:33:38.743 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:33:38.747 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:33:38.747 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:33:38.788 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:33:38.789 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:33:38.952 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:33:39.041 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:39.416 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:39.674 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:39.674 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:39.674 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:33:39.674 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:39.674 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:39.675 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:33:39.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:33:40.219 INFO:teuthology.orchestra.run.smithi195.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 2.3 MB/s | 52 MB 00:22 2024-03-25T19:33:40.220 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:40.220 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.3 MB/s | 52 MB 00:22 2024-03-25T19:33:40.221 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:33:40.225 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:33:40.225 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:33:40.442 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:02:53 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:33:40.571 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:33:40.572 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:33:40.594 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:33:40.594 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:40.594 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:33:40.594 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:40.594 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:33:40.594 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:33:40.595 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:40.595 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:33:40.595 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:40.595 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:33:40.595 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:40.595 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 771 k 2024-03-25T19:33:40.595 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.7 M 2024-03-25T19:33:40.596 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:33:41.744 INFO:teuthology.orchestra.run.smithi107.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 671 kB/s | 771 kB 00:01 2024-03-25T19:33:41.744 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:41.745 INFO:teuthology.orchestra.run.smithi107.stdout:Total 670 kB/s | 771 kB 00:01 2024-03-25T19:33:41.745 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:33:41.749 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:33:41.749 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:33:41.793 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:33:41.794 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:33:41.977 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:33:42.067 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:42.455 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:42.715 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:42.715 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:42.715 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:33:42.715 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:42.715 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:42.716 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:33:42.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:33:43.492 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:02:56 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:33:43.641 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:33:43.641 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:43.641 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:33:43.641 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:43.641 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:33:43.641 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:33:43.641 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:43.641 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:33:43.642 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:43.642 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:33:43.642 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:43.642 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.1 M 2024-03-25T19:33:43.642 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 4.3 M 2024-03-25T19:33:43.642 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:33:46.430 INFO:teuthology.orchestra.run.smithi107.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 413 kB/s | 1.1 MB 00:02 2024-03-25T19:33:46.431 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:46.431 INFO:teuthology.orchestra.run.smithi107.stdout:Total 413 kB/s | 1.1 MB 00:02 2024-03-25T19:33:46.432 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:33:46.436 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:33:46.436 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:33:46.480 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:33:46.480 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:33:46.555 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:33:46.660 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:46.734 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:33:46.795 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:47.171 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:47.427 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:33:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:47.427 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:47.427 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:33:47.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:33:48.216 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:03:01 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:33:48.366 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:33:48.366 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:33:48.367 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:48.367 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:33:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:33:48.367 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:48.367 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:33:48.367 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:48.369 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:33:48.369 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:48.369 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 337 k 2024-03-25T19:33:48.369 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.0 M 2024-03-25T19:33:48.369 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:33:48.802 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:49.085 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:49.085 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:49.085 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:33:49.085 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:49.085 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:49.085 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:33:49.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:33:49.625 INFO:teuthology.orchestra.run.smithi107.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 268 kB/s | 337 kB 00:01 2024-03-25T19:33:49.626 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:49.626 INFO:teuthology.orchestra.run.smithi107.stdout:Total 268 kB/s | 337 kB 00:01 2024-03-25T19:33:49.626 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:33:49.630 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:33:49.630 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:33:49.671 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:33:49.672 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:33:49.809 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:33:49.842 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:03:03 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:33:49.915 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:49.991 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:33:49.991 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:49.991 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:33:49.991 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:49.991 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:33:49.991 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:33:49.991 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:49.991 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:33:49.992 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:49.992 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:33:49.992 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:49.992 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 615 k 2024-03-25T19:33:49.992 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 2.1 M 2024-03-25T19:33:49.992 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:33:50.299 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:50.569 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:50.569 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:50.570 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:33:50.570 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:50.570 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:50.570 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:33:50.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:33:51.342 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:03:04 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:33:51.490 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:33:51.490 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:51.490 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:33:51.490 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:51.491 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:33:51.491 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:33:51.491 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:51.491 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:33:51.491 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:51.491 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:33:51.491 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:51.492 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 509 k 2024-03-25T19:33:51.492 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.5 M 2024-03-25T19:33:51.492 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:33:51.839 INFO:teuthology.orchestra.run.smithi195.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 333 kB/s | 615 kB 00:01 2024-03-25T19:33:51.839 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:51.840 INFO:teuthology.orchestra.run.smithi195.stdout:Total 333 kB/s | 615 kB 00:01 2024-03-25T19:33:51.840 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:33:51.844 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:33:51.844 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:33:51.886 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:33:51.886 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:33:52.053 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:33:52.147 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:52.462 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 525 kB/s | 509 kB 00:00 2024-03-25T19:33:52.465 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:52.465 INFO:teuthology.orchestra.run.smithi107.stdout:Total 524 kB/s | 509 kB 00:00 2024-03-25T19:33:52.465 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:33:52.466 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:33:52.466 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:33:52.506 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:33:52.506 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:33:52.515 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:52.670 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:33:52.769 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:52.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:52.793 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:52.793 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:33:52.793 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:52.793 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:52.793 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:33:52.995 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:33:53.155 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:53.428 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:53.428 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:53.428 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:33:53.428 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:53.428 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:53.429 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:33:53.561 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:03:07 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:33:53.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:33:53.714 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:33:53.714 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 771 k 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 2.7 M 2024-03-25T19:33:53.715 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:33:54.179 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:03:07 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:33:54.328 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:33:54.328 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:54.328 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:33:54.329 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:54.329 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:33:54.329 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:33:54.329 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:54.329 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:33:54.329 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:33:54.329 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:33:54.329 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:33:54.330 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 52 M 2024-03-25T19:33:54.330 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 298 M 2024-03-25T19:33:54.330 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:33:54.800 INFO:teuthology.orchestra.run.smithi195.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 711 kB/s | 771 kB 00:01 2024-03-25T19:33:54.801 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:54.801 INFO:teuthology.orchestra.run.smithi195.stdout:Total 710 kB/s | 771 kB 00:01 2024-03-25T19:33:54.801 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:33:54.805 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:33:54.805 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:33:54.848 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:33:54.849 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:33:55.052 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:33:55.149 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:55.537 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:55.811 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:55.811 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:55.811 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:33:55.811 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:55.811 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:55.811 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:33:56.018 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:33:56.581 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:03:10 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:33:56.729 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:33:56.730 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:56.730 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:33:56.730 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:56.730 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:33:56.731 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:33:56.731 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:56.731 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:33:56.731 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:33:56.731 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:33:56.731 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:56.731 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 1.1 M 2024-03-25T19:33:56.731 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 4.3 M 2024-03-25T19:33:56.731 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:33:58.503 INFO:teuthology.orchestra.run.smithi195.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 649 kB/s | 1.1 MB 00:01 2024-03-25T19:33:58.504 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:58.504 INFO:teuthology.orchestra.run.smithi195.stdout:Total 649 kB/s | 1.1 MB 00:01 2024-03-25T19:33:58.505 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:33:58.509 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:33:58.509 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:33:58.553 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:33:58.553 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:33:58.773 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:33:58.887 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:59.282 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:59.559 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:59.559 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:59.559 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:33:59.559 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:59.559 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:33:59.560 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:33:59.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:34:00.329 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:03:14 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:34:00.479 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:34:00.480 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:00.480 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:34:00.480 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:00.480 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:34:00.480 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:34:00.480 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:00.480 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:34:00.480 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:00.480 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:34:00.481 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:00.481 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 337 k 2024-03-25T19:34:00.481 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 1.0 M 2024-03-25T19:34:00.481 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:34:01.273 INFO:teuthology.orchestra.run.smithi195.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 426 kB/s | 337 kB 00:00 2024-03-25T19:34:01.274 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:01.274 INFO:teuthology.orchestra.run.smithi195.stdout:Total 425 kB/s | 337 kB 00:00 2024-03-25T19:34:01.274 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:34:01.278 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:34:01.278 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:34:01.318 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:34:01.319 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:34:01.458 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:34:01.565 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:34:01.912 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:34:02.192 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:34:02.192 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:02.192 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:34:02.192 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:02.192 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:02.192 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:34:02.399 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:34:02.963 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:03:16 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:34:03.113 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:34:03.114 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:03.114 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:34:03.114 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:03.114 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:34:03.114 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:34:03.114 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:03.114 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:34:03.115 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:03.115 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:34:03.115 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:03.115 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 509 k 2024-03-25T19:34:03.115 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 2.5 M 2024-03-25T19:34:03.115 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:34:04.009 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 569 kB/s | 509 kB 00:00 2024-03-25T19:34:04.010 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:04.010 INFO:teuthology.orchestra.run.smithi195.stdout:Total 568 kB/s | 509 kB 00:00 2024-03-25T19:34:04.010 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:34:04.014 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:34:04.014 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:34:04.053 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:34:04.054 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:34:04.211 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:34:04.308 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:04.667 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:04.948 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:04.949 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:04.949 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:34:04.949 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:04.949 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:04.949 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:34:05.141 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:34:05.712 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:03:19 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:34:05.861 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:34:05.862 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:05.862 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:34:05.862 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:05.862 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:34:05.862 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:34:05.862 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:05.862 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:34:05.863 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:05.863 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:34:05.863 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:05.863 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 52 M 2024-03-25T19:34:05.863 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 298 M 2024-03-25T19:34:05.863 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:34:32.585 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 52 MB 00:26 2024-03-25T19:34:32.585 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:32.586 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.9 MB/s | 52 MB 00:26 2024-03-25T19:34:32.586 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:34:32.589 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:34:32.589 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:34:32.927 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:34:32.927 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:34:38.850 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:34:38.949 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:40.969 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:41.251 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:34:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:41.251 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:34:41.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:34:41.995 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 1:03:55 ago on Mon 25 Mar 2024 06:30:46 PM UTC. 2024-03-25T19:34:42.144 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-25T19:34:42.144 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-25T19:34:42.144 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-25T19:34:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:34:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-25T19:34:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-25T19:34:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-25T19:34:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 1.3 M 2024-03-25T19:34:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 5.5 M 2024-03-25T19:34:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-25T19:34:43.422 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.0 MB/s | 52 MB 00:49 2024-03-25T19:34:43.423 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:43.424 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.0 MB/s | 52 MB 00:49 2024-03-25T19:34:43.424 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:34:43.427 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:34:43.428 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:34:43.763 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:34:43.764 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:34:44.378 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 574 kB/s | 1.3 MB 00:02 2024-03-25T19:34:44.379 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:44.379 INFO:teuthology.orchestra.run.smithi195.stdout:Total 574 kB/s | 1.3 MB 00:02 2024-03-25T19:34:44.379 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-25T19:34:44.383 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-25T19:34:44.383 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-25T19:34:44.428 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-25T19:34:44.429 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-25T19:34:44.678 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-25T19:34:44.785 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:34:45.151 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:34:45.434 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:34:45.434 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:45.434 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-25T19:34:45.434 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:45.434 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:45.434 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-25T19:34:45.662 DEBUG:teuthology.parallel:result is None 2024-03-25T19:34:49.699 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:34:49.804 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:51.861 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:52.119 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:52.119 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:52.119 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:34:52.119 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:52.119 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:52.119 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:34:52.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:34:52.869 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 1:04:05 ago on Mon 25 Mar 2024 06:30:47 PM UTC. 2024-03-25T19:34:53.018 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-25T19:34:53.018 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:34:53.019 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-25T19:34:53.019 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:34:53.019 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-25T19:34:53.019 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:34:53.019 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:53.019 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-25T19:34:53.019 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-25T19:34:53.019 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-25T19:34:53.019 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:53.020 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.3 M 2024-03-25T19:34:53.020 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 5.5 M 2024-03-25T19:34:53.020 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-25T19:34:54.299 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 1.3 MB 00:01 2024-03-25T19:34:54.299 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:54.300 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.0 MB/s | 1.3 MB 00:01 2024-03-25T19:34:54.300 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-25T19:34:54.304 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-25T19:34:54.304 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-25T19:34:54.349 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-25T19:34:54.350 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-25T19:34:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-25T19:34:54.706 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:34:55.140 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:34:55.387 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:34:55.387 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:55.387 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-25T19:34:55.388 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:55.388 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:55.388 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-25T19:34:55.598 DEBUG:teuthology.parallel:result is None 2024-03-25T19:34:55.599 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:34:55.744 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:34:55.764 INFO:teuthology.orchestra.run.smithi107.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:34:55.765 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:34:55.765 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:34:55.766 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:34:55.977 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:34:56.001 INFO:teuthology.orchestra.run.smithi195.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:34:56.001 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:34:56.002 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:34:56.003 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:34:56.003 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:34:56.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:34:56.031 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:34:56.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:34:56.072 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:34:56.072 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:34:56.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:34:56.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:34:56.166 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:34:56.166 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:34:56.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:34:56.267 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:34:56.267 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:34:56.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:34:56.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:34:56.361 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:34:56.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:34:56.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:34:56.460 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:34:56.460 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:34:56.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:34:56.487 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:34:56.554 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:34:56.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:34:56.583 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:34:56.651 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:34:56.735 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:34:56.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2024-03-25T19:34:56.737 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /var/log/ceph 2024-03-25T19:34:56.765 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:34:56.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:34:56.805 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:34:56.832 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:34:56.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:34:56.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:34:56.909 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:34:56.909 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:34:56.910 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621331', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBLuZd9QyE9I0h1TLT3GEMuC6haZ0Y7lzY5OJwwzJRByM7qKFM0qvQe2PpjZotyn3JxWXAqMDyg7SYU3euO/Z1k=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNImLuourlybiOWkqsOsHAugVHoQs9Bh5WVVf1tuCtJ0N1zw/lTL5XFajoWPCm4o+1QJQL1y04HdVYF9xa+skvU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:34:56.910 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:34:56.937 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:34:56.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:34:56.993 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:34:57.021 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:34:57.021 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:34:57.072 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:34:57.072 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-03-25T19:34:57.127 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:34:57.127 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:57.127 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T19:34:57.127 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:57.127 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:57.127 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-25 18:35:45.187647153 +0000 2024-03-25T19:34:57.127 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-25 18:29:48.915168897 +0000 2024-03-25T19:34:57.127 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-25 18:29:48.915168897 +0000 2024-03-25T19:34:57.127 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-25 18:29:48.915168897 +0000 2024-03-25T19:34:57.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:34:57.192 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-25T19:34:57.193 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-25T19:34:57.193 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000265578 s, 1.9 MB/s 2024-03-25T19:34:57.194 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:34:57.250 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-03-25T19:34:57.305 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:34:57.305 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:57.305 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T19:34:57.305 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:57.305 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:57.305 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-25 18:35:45.188647137 +0000 2024-03-25T19:34:57.306 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-25 18:29:49.243163460 +0000 2024-03-25T19:34:57.306 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-25 18:29:49.243163460 +0000 2024-03-25T19:34:57.306 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-25 18:29:49.243163460 +0000 2024-03-25T19:34:57.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:34:57.370 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-25T19:34:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-25T19:34:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000187244 s, 2.7 MB/s 2024-03-25T19:34:57.372 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:34:57.428 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-03-25T19:34:57.483 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:34:57.483 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:57.483 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:34:57.483 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:57.483 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:57.483 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-25 18:35:45.188647137 +0000 2024-03-25T19:34:57.484 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-25 18:29:49.579157890 +0000 2024-03-25T19:34:57.484 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-25 18:29:49.579157890 +0000 2024-03-25T19:34:57.484 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-25 18:29:49.579157890 +0000 2024-03-25T19:34:57.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:34:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-25T19:34:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-25T19:34:57.549 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000188167 s, 2.7 MB/s 2024-03-25T19:34:57.550 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:34:57.606 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-03-25T19:34:57.661 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:34:57.661 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:57.661 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T19:34:57.661 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:57.661 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:57.661 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-25 18:35:45.188647137 +0000 2024-03-25T19:34:57.661 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-25 18:29:49.914152337 +0000 2024-03-25T19:34:57.661 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-25 18:29:49.914152337 +0000 2024-03-25T19:34:57.662 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-25 18:29:49.914152337 +0000 2024-03-25T19:34:57.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:34:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-25T19:34:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-25T19:34:57.727 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000194322 s, 2.6 MB/s 2024-03-25T19:34:57.728 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:34:57.783 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:57.784 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:34:57.784 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:34:57.802 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:34:57.802 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2024-03-25T19:34:57.860 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:34:57.860 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:57.860 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T19:34:57.860 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:57.860 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:57.860 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-25 18:36:31.777908147 +0000 2024-03-25T19:34:57.860 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-25 18:29:48.982210407 +0000 2024-03-25T19:34:57.860 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-25 18:29:48.982210407 +0000 2024-03-25T19:34:57.861 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-25 18:29:48.982210407 +0000 2024-03-25T19:34:57.861 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:34:57.929 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-25T19:34:57.929 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-25T19:34:57.929 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000110763 s, 4.6 MB/s 2024-03-25T19:34:57.930 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:34:57.988 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2024-03-25T19:34:58.046 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:34:58.046 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:58.046 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T19:34:58.046 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:58.046 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:58.046 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-25 18:36:31.777908147 +0000 2024-03-25T19:34:58.046 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-25 18:29:49.294204204 +0000 2024-03-25T19:34:58.046 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-25 18:29:49.294204204 +0000 2024-03-25T19:34:58.046 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-25 18:29:49.294204204 +0000 2024-03-25T19:34:58.047 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:34:58.114 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-25T19:34:58.114 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-25T19:34:58.114 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000108043 s, 4.7 MB/s 2024-03-25T19:34:58.115 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:34:58.172 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2024-03-25T19:34:58.227 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:34:58.227 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:58.227 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:34:58.228 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:58.228 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:58.228 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-25 18:36:31.778908128 +0000 2024-03-25T19:34:58.228 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-25 18:29:49.636197404 +0000 2024-03-25T19:34:58.228 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-25 18:29:49.636197404 +0000 2024-03-25T19:34:58.228 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-25 18:29:49.636197404 +0000 2024-03-25T19:34:58.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:34:58.293 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-25T19:34:58.293 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-25T19:34:58.293 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000119188 s, 4.3 MB/s 2024-03-25T19:34:58.295 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:34:58.351 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2024-03-25T19:34:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:34:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:58.406 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T19:34:58.406 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:58.406 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:58.406 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-25 18:36:31.778908128 +0000 2024-03-25T19:34:58.406 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-25 18:29:49.949191181 +0000 2024-03-25T19:34:58.406 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-25 18:29:49.949191181 +0000 2024-03-25T19:34:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-25 18:29:49.949191181 +0000 2024-03-25T19:34:58.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:34:58.471 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-25T19:34:58.471 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-25T19:34:58.471 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 8.0771e-05 s, 6.3 MB/s 2024-03-25T19:34:58.472 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:34:58.527 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:58.527 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:34:58.528 INFO:tasks.ceph:Generating config... 2024-03-25T19:34:58.529 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:34:58.529 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:34:58.529 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:34:58.529 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:34:58.529 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:34:58.529 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:34:58.529 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:34:58.529 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:34:58.529 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:34:58.529 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:34:58.529 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:34:58.530 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:34:58.530 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:34:58.530 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:34:58.530 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:34:58.531 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:34:58.532 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:34:58.532 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:34:58.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:34:58.587 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:34:58.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:34:58.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:34:58.662 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.107:6789]'), ('mon.c', '[v1:172.21.15.107:6790]'), ('mon.b', '[v1:172.21.15.195:6789]')] 2024-03-25T19:34:58.663 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.107:6789],[v1:172.21.15.107:6790],[v1:172.21.15.195:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:34:58.664 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:34:58.664 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:34:58.718 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.107:6789]' --addv c '[v1:172.21.15.107:6790]' --addv b '[v1:172.21.15.195:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:58.802 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: generated fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = quincy 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 0 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:min_mon_release 17 (quincy) 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:election_strategy: 3 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:34:58.803 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:34:58.805 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:34:58.819 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d993d317-0537-43ba-81bc-c1cea4d4fd58... 2024-03-25T19:34:58.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:34:58.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:34:58.909 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2024-03-25T19:34:58.909 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2024-03-25T19:34:58.909 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:34:58.909 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = cephx 2024-03-25T19:34:58.909 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.909 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2024-03-25T19:34:58.909 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:34:58.910 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: mon_allow_pool_size_one = true 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.911 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: mon health detail to clog = false 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = "[v1:172.21.15.107:6789],[v1:172.21.15.107:6790],[v1:172.21.15.195:6789]" 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: mon client directed command retry = 5 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: mon election default strategy = 3 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: ms bind msgr2 = False 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject socket failures = 5000 2024-03-25T19:34:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: ms type = async 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: osd heartbeat grace = 80 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 600 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2024-03-25T19:34:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug trim objects = true 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:34:58.914 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: bdev async discard = True 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: bdev enable discard = True 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore allocator = stupid 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore zero block detection = True 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 1 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 1 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: debug monc = 20 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 20 2024-03-25T19:34:58.915 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 10 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: osd class default list = * 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: osd class load list = * 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: osd client watch timeout = 120 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify cached snaps = True 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify missing on start = True 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: osd fast shutdown = False 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: osd max markdown count = 1000 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock override recovery settings = True 2024-03-25T19:34:58.916 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue = debug_random 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue cut off = debug_random 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: osd shutdown pgref assert = True 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.917 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 20 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:34:58.918 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: # 1m isn't quite enough 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: mon min osdmap epochs = 50 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd crush smoke test = False 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune interval = 2 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune min = 15 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: mon scrub interval = 300 2024-03-25T19:34:58.919 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on pool no app = False 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout: paxos service trim min = 10 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout: debug monc = 20 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout: debug objecter = 20 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout: debug rados = 20 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2024-03-25T19:34:58.920 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.c] 2024-03-25T19:34:58.921 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.b] 2024-03-25T19:34:58.921 INFO:teuthology.orchestra.run.smithi195.stdout:[global] 2024-03-25T19:34:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: chdir = "" 2024-03-25T19:34:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:34:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: auth supported = cephx 2024-03-25T19:34:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: filestore xattr use omap = true 2024-03-25T19:34:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: mon clock drift allowed = 1.000 2024-03-25T19:34:58.921 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: osd crush chooseleaf type = 0 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: auth debug = true 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on old message = true 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on bug = true 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: mon pg warn max object skew = 0 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default size = 2 2024-03-25T19:34:58.922 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow primary affinity = true 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow pg remap = true 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on no sortbitwise = false 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on too few osds = false 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: mon_allow_pool_size_one = true 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.923 INFO:teuthology.orchestra.run.smithi195.stdout: osd default data pool replay window = 5 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: mon allow pool delete = true 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: mon cluster log file level = debug 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: debug asserts on shutdown = true 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: mon health detail to clog = false 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: mon host = "[v1:172.21.15.107:6789],[v1:172.21.15.107:6790],[v1:172.21.15.195:6789]" 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: mon client directed command retry = 5 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: mon election default strategy = 3 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: ms bind msgr2 = False 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject socket failures = 5000 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: ms type = async 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: osd heartbeat grace = 80 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: fsid = d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:34:58.924 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout:[osd] 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: osd journal size = 100 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub load threshold = 5.0 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub max interval = 600 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: osd recover clone overlap = true 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: osd recovery max chunk = 1048576 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug shutdown = true 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug op order = true 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify stray on activate = true 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug trim objects = true 2024-03-25T19:34:58.925 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: osd open classes on start = true 2024-03-25T19:34:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug pg log writeout = true 2024-03-25T19:34:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:34:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: osd map max advance = 10 2024-03-25T19:34:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: journal zero on create = true 2024-03-25T19:34:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:34:58.926 INFO:teuthology.orchestra.run.smithi195.stdout: filestore apply finisher threads = 3 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: bdev debug aio = true 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug misdirected ops = true 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: bdev async discard = True 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: bdev enable discard = True 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore allocator = stupid 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore block size = 96636764160 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore fsck on mount = True 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore zero block detection = True 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluefs = 1 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluestore = 1 2024-03-25T19:34:58.927 INFO:teuthology.orchestra.run.smithi195.stdout: debug monc = 20 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: debug osd = 20 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: debug rocksdb = 10 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd full ratio = 0.9 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: osd class default list = * 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: osd class load list = * 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: osd client watch timeout = 120 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify cached snaps = True 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify missing on start = True 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: osd fast shutdown = False 2024-03-25T19:34:58.928 INFO:teuthology.orchestra.run.smithi195.stdout: osd max markdown count = 1000 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock override recovery settings = True 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: osd objectstore = bluestore 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: osd op queue = debug_random 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: osd op queue cut off = debug_random 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: osd shutdown pgref assert = True 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr] 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: debug mgr = 20 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: mgr/telemetry/nag = false 2024-03-25T19:34:58.929 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout:[mon] 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: debug paxos = 20 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: mon data avail warn = 5 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: mon mgr mkfs grace = 240 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd reporter subtree level = osd 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd prime pg temp = true 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:34:58.930 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: # 1m isn't quite enough 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: mon min osdmap epochs = 50 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd crush smoke test = False 2024-03-25T19:34:58.931 INFO:teuthology.orchestra.run.smithi195.stdout: mon osdmap full prune interval = 2 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: mon osdmap full prune min = 15 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: mon scrub interval = 300 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on pool no app = False 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: paxos service trim min = 10 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout:[client] 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: rgw cache enabled = true 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable ops log = true 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable usage log = true 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: debug monc = 20 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-03-25T19:34:58.932 INFO:teuthology.orchestra.run.smithi195.stdout: debug objecter = 20 2024-03-25T19:34:58.933 INFO:teuthology.orchestra.run.smithi195.stdout: debug rados = 20 2024-03-25T19:34:58.933 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.a] 2024-03-25T19:34:58.933 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.c] 2024-03-25T19:34:58.933 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.b] 2024-03-25T19:34:58.934 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:34:58.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:34:58.989 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:34:58.989 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:34:58.989 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:34:59.004 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:34:59.004 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:34:59.059 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2024-03-25T19:34:59.059 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:34:59.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:34:59.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:34:59.138 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:34:59.139 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:59.193 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi195.front.sepia.ceph.com 2024-03-25T19:34:59.193 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:34:59.193 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:34:59.193 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:34:59.233 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:34:59.233 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:59.287 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:34:59.287 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:34:59.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:34:59.342 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:34:59.344 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:34:59.403 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:34:59.405 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:34:59.405 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:34:59.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:34:59.447 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:34:59.462 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:34:59.505 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:34:59.520 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:34:59.521 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:34:59.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:34:59.547 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:59.547 INFO:tasks.ceph:role: osd.0 2024-03-25T19:34:59.547 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-25T19:34:59.547 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:34:59.621 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:59.622 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:59.622 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:59.622 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:59.622 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:59.622 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:59.622 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:59.622 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:59.622 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:59.622 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:35:00.134 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-03-25T19:35:00.136 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-03-25T19:35:00.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:35:00.281 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:35:00.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:35:00.384 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:35:00.384 INFO:tasks.ceph:role: osd.1 2024-03-25T19:35:00.384 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-25T19:35:00.384 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:35:00.457 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:35:00.458 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:35:00.458 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:35:00.458 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:35:00.458 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:35:00.458 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:35:00.458 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:35:00.458 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:35:00.458 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:35:00.458 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:35:00.904 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-03-25T19:35:00.906 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-03-25T19:35:00.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:35:00.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:35:01.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:35:01.089 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:35:01.089 INFO:tasks.ceph:role: osd.2 2024-03-25T19:35:01.089 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-25T19:35:01.089 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:35:01.164 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:35:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:35:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:35:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:35:01.164 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:35:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:35:01.164 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:35:01.164 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:35:01.164 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:35:01.164 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:35:01.655 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-03-25T19:35:01.657 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-03-25T19:35:01.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:35:01.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:35:01.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:35:01.880 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:35:01.880 INFO:tasks.ceph:role: osd.3 2024-03-25T19:35:01.880 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-25T19:35:01.880 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:35:01.955 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:35:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:35:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:35:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:35:01.955 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:35:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:35:01.955 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:35:01.955 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:35:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:35:01.955 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:35:02.450 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-03-25T19:35:02.452 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-03-25T19:35:02.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:35:02.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:35:02.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:02.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:02.705+0000 7f3e90814640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:35:02.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:02.705+0000 7f3e90814640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:35:02.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:02.705+0000 7f3e90814640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:35:02.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:02.705+0000 7f3e90814640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:35:02.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:02.705+0000 7f3e90814640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:35:02.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:02.706+0000 7f3e90814640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:35:05.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:35:05.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:05.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:05.680+0000 7f6000921640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:35:05.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:05.680+0000 7f6000921640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:35:05.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:05.680+0000 7f6000921640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:35:05.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:05.680+0000 7f6000921640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:35:05.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:05.680+0000 7f6000921640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:35:05.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:05.681+0000 7f6000921640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:35:08.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:35:08.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:08.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:08.638+0000 7ff8391fd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:35:08.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:08.638+0000 7ff8391fd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:35:08.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:08.638+0000 7ff8391fd640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:35:08.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:08.638+0000 7ff8391fd640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:35:08.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:08.638+0000 7ff8391fd640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:35:08.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:08.639+0000 7ff8391fd640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:35:11.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:35:11.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:11.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:11.615+0000 7ff65e9a8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:35:11.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:11.615+0000 7ff65e9a8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:35:11.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:11.615+0000 7ff65e9a8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:35:11.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:11.615+0000 7ff65e9a8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:35:11.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:11.615+0000 7ff65e9a8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:35:11.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:11.616+0000 7ff65e9a8640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:35:14.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:35:14.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:35:14.521 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:35:14.521 INFO:tasks.ceph:role: osd.4 2024-03-25T19:35:14.521 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-25T19:35:14.521 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:35:14.591 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:35:14.592 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:35:14.592 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:35:14.592 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:35:14.592 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:35:14.592 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:35:14.592 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:35:14.592 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:35:14.592 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:35:14.592 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:35:14.624 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-03-25T19:35:14.626 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-03-25T19:35:14.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:35:14.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:35:14.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:35:14.838 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:35:14.838 INFO:tasks.ceph:role: osd.5 2024-03-25T19:35:14.838 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-25T19:35:14.838 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:35:14.904 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:35:14.904 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:35:14.904 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:35:14.904 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:35:14.904 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:35:14.904 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:35:14.905 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:35:14.905 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:35:14.905 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:35:14.905 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:35:14.937 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-03-25T19:35:14.939 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-03-25T19:35:14.939 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:35:15.009 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:35:15.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:35:15.151 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:35:15.151 INFO:tasks.ceph:role: osd.6 2024-03-25T19:35:15.151 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-25T19:35:15.151 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:35:15.217 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:35:15.217 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:35:15.217 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:35:15.217 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:35:15.217 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:35:15.218 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:35:15.218 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:35:15.218 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:35:15.218 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:35:15.218 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:35:15.250 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-03-25T19:35:15.252 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-03-25T19:35:15.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:35:15.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:35:15.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:35:15.464 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:35:15.464 INFO:tasks.ceph:role: osd.7 2024-03-25T19:35:15.464 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-25T19:35:15.464 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:35:15.530 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:35:15.531 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:35:15.531 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:35:15.531 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:35:15.531 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:35:15.531 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:35:15.531 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:35:15.531 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:35:15.531 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:35:15.531 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:35:15.563 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-03-25T19:35:15.565 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-03-25T19:35:15.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:35:15.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:35:15.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:15.809 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:15.809+0000 7fe4c5ebd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:35:15.810 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:15.810+0000 7fe4c5ebd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:35:15.810 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:15.810+0000 7fe4c5ebd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:35:15.810 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:15.810+0000 7fe4c5ebd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:35:15.810 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:15.810+0000 7fe4c5ebd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:35:15.810 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:15.810+0000 7fe4c5ebd640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:35:18.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:35:18.688 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:18.779 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:18.779+0000 7f8878a18640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:35:18.779 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:18.779+0000 7f8878a18640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:35:18.779 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:18.779+0000 7f8878a18640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:35:18.779 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:18.779+0000 7f8878a18640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:35:18.779 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:18.779+0000 7f8878a18640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:35:18.780 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:18.780+0000 7f8878a18640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:35:21.617 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:35:21.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:21.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:21.739+0000 7fc92d5ac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:35:21.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:21.740+0000 7fc92d5ac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:35:21.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:21.740+0000 7fc92d5ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:35:21.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:21.740+0000 7fc92d5ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:35:21.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:21.740+0000 7fc92d5ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:35:21.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:21.740+0000 7fc92d5ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:35:24.585 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:35:24.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:24.708 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:24.708+0000 7f362d80d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:35:24.708 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:24.708+0000 7f362d80d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:35:24.708 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:24.708+0000 7f362d80d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:35:24.708 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:24.708+0000 7f362d80d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:35:24.708 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:24.708+0000 7f362d80d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:35:24.708 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:24.708+0000 7f362d80d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:35:27.554 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:35:27.585 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:35:27.585 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:35:27.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:35:27.617 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:35:27.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:35:27.688 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:35:27.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:35:27.758 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:35:27.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:35:27.826 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:35:27.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:35:27.894 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:35:27.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:35:27.926 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:35:27.926 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:35:27.997 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:35:27.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:35:28.067 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:35:28.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:35:28.136 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:35:28.136 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:35:28.205 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:35:28.205 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:35:28.221 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:35:28.221 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:35:28.261 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:35:28.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:35:28.263 DEBUG:teuthology.orchestra.run.smithi195:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:35:28.304 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.y] 2024-03-25T19:35:28.304 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDj0QFmPXh8FBAAIiPUNMZWfo8b9rPxdihZRQ== 2024-03-25T19:35:28.304 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2024-03-25T19:35:28.304 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDm0QFm130XKhAA6PVP2LXlSPk4aC+1eidLGw== 2024-03-25T19:35:28.304 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDp0QFmyGOpKBAACMztBupc4Eq87fGN43wI5w== 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDs0QFm+KwnJhAAcrCMlc/KlrBnbBuPCJSeWQ== 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.3] 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDv0QFm4rW8JBAA6sfvfW4Z2TnZCxFGIGgAdw== 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.x] 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDj0QFmAYYdGBAAuCi7SBHDtPT8nXWB1sPc8w== 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.4] 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDz0QFmd2FTMBAArmwmqYIj6VObw302J0UAEQ== 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.5] 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD20QFmaxeHLhAAFIT/yBCh9yIQ5HICyGnvkQ== 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.6] 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD50QFmbRMnLBAApG/XzmKV9j40IhRMq+78SA== 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.7] 2024-03-25T19:35:28.305 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQD80QFmhBNFKhAAkdFldrho0BDvPEMoKxXRCA== 2024-03-25T19:35:28.306 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2024-03-25T19:35:28.306 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDj0QFmv2XHGhAAzJQC203ji1Y1A3P26/wlTQ== 2024-03-25T19:35:28.306 INFO:teuthology.orchestra.run.smithi107.stdout:[client.1] 2024-03-25T19:35:28.306 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDj0QFm90I0HhAA70An5SGcpfG+CW11WtBO1w== 2024-03-25T19:35:28.327 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.y] 2024-03-25T19:35:28.327 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDj0QFmPXh8FBAAIiPUNMZWfo8b9rPxdihZRQ== 2024-03-25T19:35:28.327 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.0] 2024-03-25T19:35:28.327 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDm0QFm130XKhAA6PVP2LXlSPk4aC+1eidLGw== 2024-03-25T19:35:28.327 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.1] 2024-03-25T19:35:28.327 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDp0QFmyGOpKBAACMztBupc4Eq87fGN43wI5w== 2024-03-25T19:35:28.327 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.2] 2024-03-25T19:35:28.327 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDs0QFm+KwnJhAAcrCMlc/KlrBnbBuPCJSeWQ== 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.3] 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDv0QFm4rW8JBAA6sfvfW4Z2TnZCxFGIGgAdw== 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.x] 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDj0QFmAYYdGBAAuCi7SBHDtPT8nXWB1sPc8w== 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.4] 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDz0QFmd2FTMBAArmwmqYIj6VObw302J0UAEQ== 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.5] 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD20QFmaxeHLhAAFIT/yBCh9yIQ5HICyGnvkQ== 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.6] 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD50QFmbRMnLBAApG/XzmKV9j40IhRMq+78SA== 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.7] 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD80QFmhBNFKhAAkdFldrho0BDvPEMoKxXRCA== 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout:[client.0] 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDj0QFmv2XHGhAAzJQC203ji1Y1A3P26/wlTQ== 2024-03-25T19:35:28.328 INFO:teuthology.orchestra.run.smithi195.stdout:[client.1] 2024-03-25T19:35:28.329 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDj0QFm90I0HhAA70An5SGcpfG+CW11WtBO1w== 2024-03-25T19:35:28.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:35:28.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:35:28.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:35:28.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:35:28.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.750 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.803 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:28.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:35:28.910 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:35:28.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:35:28.967 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:35:29.022 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:35:29.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:35:29.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:35:29.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:35:29.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:35:29.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:35:29.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:35:29.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:35:29.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:35:29.488 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:35:29.515 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:29.517 DEBUG:teuthology.orchestra.run.smithi195:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:29.570 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:35:29.570 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:35:29.570 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:35:29.570 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:35:29.572 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:35:29.572 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:35:29.573 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:35:29.573 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:35:29.574 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:35:29.574 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:35:29.575 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:35:29.575 DEBUG:teuthology.orchestra.run.smithi195:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:35:29.612 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:35:29.613 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:35:29.613 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:35:29.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:35:29.615 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:35:29.615 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:35:29.616 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:35:29.617 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:35:29.618 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:35:29.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:35:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.719+0000 7fddc120f640 10 monclient: get_monmap_and_config 2024-03-25T19:35:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.719+0000 7fddc120f640 10 monclient: build_initial_monmap 2024-03-25T19:35:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.719+0000 7fddc120f640 10 monclient: monmap: 2024-03-25T19:35:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2024-03-25T19:35:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-25T19:35:29.720722+0000 2024-03-25T19:35:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-25T19:35:29.720722+0000 2024-03-25T19:35:29.719 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:29.720 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:29.720 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:29.720 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:35:29.720 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:35:29.720 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:29.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.720+0000 7fddc120f640 1 Processor -- start 2024-03-25T19:35:29.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.720+0000 7fddc120f640 1 -- start start 2024-03-25T19:35:29.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.720+0000 7fddc120f640 10 monclient: init 2024-03-25T19:35:29.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.720+0000 7fddc120f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.720+0000 7fddc120f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.720+0000 7fddc120f640 10 monclient(hunting): picked mon.noname-a con 0x7fddbc0a9920 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.720+0000 7fddc120f640 10 monclient(hunting): picked mon.noname-b con 0x7fddbc0b45a0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.720+0000 7fddc120f640 10 monclient(hunting): picked mon.noname-c con 0x7fddbc146450 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.720+0000 7fddc120f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc06b100 con 0x7fddbc0a9920 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.721+0000 7fddc120f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc0b2940 con 0x7fddbc146450 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.721+0000 7fddc120f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc13fec0 con 0x7fddbc0b45a0 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.721+0000 7fddc120f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.721+0000 7fddba59c640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fddbc145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.721+0000 7fddbad9d640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fddbc13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.721+0000 7fddc120f640 10 monclient(hunting): authenticate will time out at 4522.061784s 2024-03-25T19:35:29.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.721+0000 7fddbb59e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fddbc14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:29.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.922+0000 7fddba59c640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fddbc145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:29.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.922+0000 7fddbad9d640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fddbc13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:29.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:29.922+0000 7fddbb59e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fddbc14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:30.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:30.323+0000 7fddbad9d640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fddbc13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:30.323+0000 7fddba59c640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fddbc145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:30.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:30.323+0000 7fddbb59e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fddbc14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:31.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:31.124+0000 7fddba59c640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fddbc145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:31.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:31.124+0000 7fddbad9d640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fddbc13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:31.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:31.124+0000 7fddbb59e640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fddbc14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 10 monclient(hunting): tick 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fddbc145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fddbc14dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fddbc13f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-b con 0x7fddbc0b45a0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-c con 0x7fddbc146450 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-a con 0x7fddbc0a9920 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:32.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc06b100 con 0x7fddbc0a9920 2024-03-25T19:35:32.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc0b2940 con 0x7fddbc146450 2024-03-25T19:35:32.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc13fec0 con 0x7fddbc0b45a0 2024-03-25T19:35:32.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddb8d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:32.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddbad9d640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:32.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddbb59e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fdda800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:32.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.721+0000 7fddba59c640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fdda8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.922+0000 7fddbad9d640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.922+0000 7fddbb59e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fdda800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:32.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:32.922+0000 7fddba59c640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fdda8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:33.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:33.323+0000 7fddbb59e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fdda800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:33.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:33.323+0000 7fddbad9d640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:33.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:33.323+0000 7fddba59c640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fdda8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:34.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:34.124+0000 7fddbb59e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fdda800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:34.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:34.124+0000 7fddbad9d640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:34.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:34.124+0000 7fddba59c640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fdda8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 10 monclient(hunting): tick 2024-03-25T19:35:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:35:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:35:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:35:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda80056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fdda8008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fdda800c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-c con 0x7fddbc146450 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-b con 0x7fddbc0b45a0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-a con 0x7fddbc0a9920 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda8010e20 con 0x7fddbc0a9920 2024-03-25T19:35:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc06b100 con 0x7fddbc146450 2024-03-25T19:35:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc0b2940 con 0x7fddbc0b45a0 2024-03-25T19:35:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddb8d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddbb59e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fdda8010500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddba59c640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.721+0000 7fddbad9d640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fdda800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.922+0000 7fddbb59e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fdda8010500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.922+0000 7fddba59c640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:35.922+0000 7fddbad9d640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fdda800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:36.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:36.323+0000 7fddbb59e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fdda8010500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:36.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:36.323+0000 7fddba59c640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:36.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:36.323+0000 7fddbad9d640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fdda800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:36.540 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:00:06.826 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:36.845 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:00:07.132 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:36.892 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:00:07.208 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:36.892 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:00:07.209 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:37.124+0000 7fddbb59e640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fdda8010500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6789/0 2024-03-25T19:35:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:37.124+0000 7fddba59c640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.107:6790/0 2024-03-25T19:35:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:37.124+0000 7fddbad9d640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fdda800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2024-03-25T19:35:37.213 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:00:07.529 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.214 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:00:07.530 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:38.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 10 monclient(hunting): tick 2024-03-25T19:35:38.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:35:38.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:35:38.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:35:38.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda80056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:38.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 1 -- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fdda800fdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:38.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 1 -- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fdda8010500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:38.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:38.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-b con 0x7fddbc0b45a0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:38.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-c con 0x7fddbc146450 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:38.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-a con 0x7fddbc0a9920 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:38.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc13fec0 con 0x7fddbc0a9920 2024-03-25T19:35:38.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda8010e20 con 0x7fddbc146450 2024-03-25T19:35:38.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc06b100 con 0x7fddbc0b45a0 2024-03-25T19:35:38.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.721+0000 7fddb8d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:38.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.741+0000 7fddba59c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 0x7fdda80056b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:40470/0 (socket says 172.21.15.107:40470) 2024-03-25T19:35:38.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:38.741+0000 7fddba59c640 1 -- 172.21.15.107:0/3939285584 learned_addr learned my addr 172.21.15.107:0/3939285584 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:35:41.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 10 monclient(hunting): tick 2024-03-25T19:35:41.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:35:41.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:35:41.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:35:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda8008e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 >> v1:172.21.15.195:6789/0 conn(0x7fddbc146450 legacy=0x7fdda80056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a9920 legacy=0x7fdda800c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-a con 0x7fdda800ddb0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-c con 0x7fdda80120a0 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-b con 0x7fddbc0b45a0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc0b2940 con 0x7fdda800ddb0 2024-03-25T19:35:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda8005700 con 0x7fdda80120a0 2024-03-25T19:35:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda8005930 con 0x7fddbc0b45a0 2024-03-25T19:35:41.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:41.722+0000 7fddb8d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:44.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.721+0000 7fddb8d99640 10 monclient(hunting): tick 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.721+0000 7fddb8d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.721+0000 7fddb8d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.721+0000 7fddb8d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.721+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 >> v1:172.21.15.107:6790/0 conn(0x7fddbc0b45a0 legacy=0x7fdda8005e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.721+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 >> v1:172.21.15.195:6789/0 conn(0x7fdda80120a0 legacy=0x7fdda8014490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.721+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 >> v1:172.21.15.107:6789/0 conn(0x7fdda800ddb0 legacy=0x7fdda8010220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.721+0000 7fddb8d99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.721+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-b con 0x7fdda8006510 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.721+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-a con 0x7fdda80069d0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.722+0000 7fddb8d99640 10 monclient(hunting): picked mon.noname-c con 0x7fdda80120a0 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.722+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda8005b60 con 0x7fdda80069d0 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.722+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda8005f00 con 0x7fdda80120a0 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.722+0000 7fddb8d99640 1 -- 172.21.15.107:0/3939285584 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda8006130 con 0x7fdda8006510 2024-03-25T19:35:44.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:44.722+0000 7fddb8d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:47.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.159+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053154568 0 0) 0x7fdda8006130 con 0x7fdda8006510 2024-03-25T19:35:47.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.159+0000 7fddb9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:47.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.159+0000 7fddb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:47.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.159+0000 7fddb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:47.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.159+0000 7fddb9d9b640 10 monclient(hunting): my global_id is 4103 2024-03-25T19:35:47.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.159+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda4003520 con 0x7fdda8006510 2024-03-25T19:35:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.195+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672581180 0 0) 0x7fdda8005f00 con 0x7fdda80120a0 2024-03-25T19:35:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.195+0000 7fddb9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.195+0000 7fddb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.195+0000 7fddb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.195+0000 7fddb9d9b640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:35:47.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.195+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda8006130 con 0x7fdda80120a0 2024-03-25T19:35:47.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.203+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385479278 0 0) 0x7fdda8005b60 con 0x7fdda80069d0 2024-03-25T19:35:47.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.203+0000 7fddb9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:47.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.203+0000 7fddb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:47.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.203+0000 7fddb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:47.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.203+0000 7fddb9d9b640 10 monclient(hunting): my global_id is 4104 2024-03-25T19:35:47.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.203+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda8005f00 con 0x7fdda80069d0 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1471758350 0 0) 0x7fdda4003520 con 0x7fdda8006510 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 >> v1:172.21.15.195:6789/0 conn(0x7fdda80120a0 legacy=0x7fdda800be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 >> v1:172.21.15.107:6789/0 conn(0x7fdda80069d0 legacy=0x7fdda800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 1 monclient: found mon.noname-b 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda8006360 con 0x7fdda8006510 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fddac003ee0 con 0x7fdda8006510 2024-03-25T19:35:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddb9d9b640 10 monclient: dump: 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddc120f640 5 monclient: authenticate success, global_id 4103 2024-03-25T19:35:47.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.215+0000 7fddc120f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddac0040b0 con 0x7fdda8006510 2024-03-25T19:35:47.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddb9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddb9d9b640 1 -- 172.21.15.107:0/3939285584 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fddac0044c0 con 0x7fdda8006510 2024-03-25T19:35:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddb9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddb9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddb9d9b640 10 monclient: dump: 2024-03-25T19:35:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-25T19:35:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:47.298 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:47.299 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:47.299 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:47.299 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:47.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddc120f640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:47.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddc120f640 4 set_mon_vals no callback set 2024-03-25T19:35:47.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddc120f640 10 monclient: shutdown 2024-03-25T19:35:47.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddc120f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:47.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddc120f640 1 -- 172.21.15.107:0/3939285584 >> v1:172.21.15.107:6790/0 conn(0x7fdda8006510 legacy=0x7fdda8008e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:47.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddc120f640 1 -- 172.21.15.107:0/3939285584 shutdown_connections 2024-03-25T19:35:47.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddc120f640 1 -- 172.21.15.107:0/3939285584 >> 172.21.15.107:0/3939285584 conn(0x7fddbc0b7f40 msgr2=0x7fddbc0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:47.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.298+0000 7fddc120f640 1 -- 172.21.15.107:0/3939285584 shutdown_connections 2024-03-25T19:35:47.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.299+0000 7fddc120f640 1 -- 172.21.15.107:0/3939285584 wait complete. 2024-03-25T19:35:47.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.299+0000 7fddc120f640 10 monclient: build_initial_monmap 2024-03-25T19:35:47.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.299+0000 7fddc120f640 10 monclient: monmap: 2024-03-25T19:35:47.302 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2024-03-25T19:35:47.303 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:47.303 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 0.000000 2024-03-25T19:35:47.303 INFO:teuthology.orchestra.run.smithi107.stderr:created 0.000000 2024-03-25T19:35:47.303 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:47.303 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-25T19:35:47.303 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:47.303 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:35:47.303 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:35:47.303 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:47.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.300+0000 7fddc120f640 1 librados: starting msgr at 2024-03-25T19:35:47.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.300+0000 7fddc120f640 1 librados: starting objecter 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.300+0000 7fddc120f640 1 Processor -- start 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.300+0000 7fddc120f640 1 -- start start 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.300+0000 7fddc120f640 1 librados: setting wanted keys 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.300+0000 7fddc120f640 1 librados: calling monclient init 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.300+0000 7fddc120f640 10 monclient: init 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.301+0000 7fddc120f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.301+0000 7fddc120f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.301+0000 7fddc120f640 10 monclient(hunting): picked mon.noname-b con 0x7fdda8006510 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.301+0000 7fddc120f640 10 monclient(hunting): picked mon.noname-a con 0x7fdda80069d0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.301+0000 7fddc120f640 10 monclient(hunting): picked mon.noname-c con 0x7fdda80120a0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.301+0000 7fddc120f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc142e30 con 0x7fdda80069d0 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.301+0000 7fddc120f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc0acd40 con 0x7fdda8006510 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.301+0000 7fddc120f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc143080 con 0x7fdda80120a0 2024-03-25T19:35:47.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.301+0000 7fddc120f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:47.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.301+0000 7fddc120f640 10 monclient(hunting): authenticate will time out at 4539.642169s 2024-03-25T19:35:47.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.303+0000 7fddba59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdda80069d0 0x7fddbc1d7410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58586/0 (socket says 172.21.15.107:58586) 2024-03-25T19:35:47.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.303+0000 7fddba59c640 1 -- 172.21.15.107:0/225107976 learned_addr learned my addr 172.21.15.107:0/225107976 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:35:47.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.304+0000 7fdda37fe640 10 client.?.objecter ms_handle_connect 0x7fdda80069d0 2024-03-25T19:35:47.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.304+0000 7fdda37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:47.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.311+0000 7fdda37fe640 10 client.?.objecter ms_handle_connect 0x7fdda80120a0 2024-03-25T19:35:47.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.311+0000 7fdda37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:47.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.315+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52194034 0 0) 0x7fddbc143080 con 0x7fdda80120a0 2024-03-25T19:35:47.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.315+0000 7fdda37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:47.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.315+0000 7fdda37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:47.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.315+0000 7fdda37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:47.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.315+0000 7fdda37fe640 10 monclient(hunting): my global_id is 4112 2024-03-25T19:35:47.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.315+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda8005b60 con 0x7fdda80120a0 2024-03-25T19:35:47.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.323+0000 7fdda37fe640 10 client.?.objecter ms_handle_connect 0x7fdda8006510 2024-03-25T19:35:47.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.323+0000 7fdda37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:47.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.332+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1754980542 0 0) 0x7fdda8005b60 con 0x7fdda80120a0 2024-03-25T19:35:47.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.332+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddbc143080 con 0x7fdda80120a0 2024-03-25T19:35:47.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.332+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fddb4004590 con 0x7fdda80120a0 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.332+0000 7fdda37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.332+0000 7fdda37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.332+0000 7fdda37fe640 10 monclient(hunting): dump: 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.332+0000 7fdda37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.395+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500922522 0 0) 0x7fddbc0acd40 con 0x7fdda8006510 2024-03-25T19:35:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.395+0000 7fdda37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.395+0000 7fdda37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.395+0000 7fdda37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.395+0000 7fdda37fe640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:35:47.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.395+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda8005b60 con 0x7fdda8006510 2024-03-25T19:35:47.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.406+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591731133 0 0) 0x7fddbc142e30 con 0x7fdda80069d0 2024-03-25T19:35:47.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.406+0000 7fdda37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:47.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.406+0000 7fdda37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:47.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.406+0000 7fdda37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:47.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.406+0000 7fdda37fe640 10 monclient(hunting): my global_id is 4110 2024-03-25T19:35:47.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.406+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddbc0acd40 con 0x7fdda80069d0 2024-03-25T19:35:47.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1664385732 0 0) 0x7fddbc143080 con 0x7fdda80120a0 2024-03-25T19:35:47.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 >> v1:172.21.15.195:6789/0 conn(0x7fdda8006510 legacy=0x7fddbc138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:47.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 >> v1:172.21.15.107:6789/0 conn(0x7fdda80069d0 legacy=0x7fddbc1d7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:47.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fdda37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:47.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fdda37fe640 1 monclient: found mon.c 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fdda37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fdda37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc142ba0 con 0x7fdda80120a0 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fdda37fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fdda37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fdda37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fddc120f640 5 monclient: authenticate success, global_id 4112 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fddc120f640 10 monclient: _renew_subs 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fddc120f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fddc120f640 1 -- 172.21.15.107:0/225107976 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddbc138a30 con 0x7fdda80120a0 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fddc120f640 10 client.4112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fddc120f640 10 monclient: _renew_subs 2024-03-25T19:35:47.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fddc120f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fddc120f640 1 -- 172.21.15.107:0/225107976 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc0abb30 con 0x7fdda80120a0 2024-03-25T19:35:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.416+0000 7fddc120f640 1 librados: init done 2024-03-25T19:35:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.417+0000 7fddc120f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.417+0000 7fddc120f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:47.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.417+0000 7fddc120f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:47.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.417+0000 7fddc120f640 1 -- 172.21.15.107:0/225107976 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddbc143210 con 0x7fdda80120a0 2024-03-25T19:35:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddb40049a0 con 0x7fdda80120a0 2024-03-25T19:35:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fddb4004db0 con 0x7fdda80120a0 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 10 monclient: dump: 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:47.645 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fddb4005360 con 0x7fdda80120a0 2024-03-25T19:35:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 4212583960 0 0) 0x7fddb4006890 con 0x7fdda80120a0 2024-03-25T19:35:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 10 client.4112.objecter ms_dispatch 0x7fddbc000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:35:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 3 client.4112.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:35:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fddb959a640 4 set_mon_vals no callback set 2024-03-25T19:35:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 3 client.4112.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:35:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 20 client.4112.objecter dump_active .. 0 homeless 2024-03-25T19:35:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fddb4018e00 con 0x7fdda80120a0 2024-03-25T19:35:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.643+0000 7fdda37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:47.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.715+0000 7fddc120f640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:35:47.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.715+0000 7fddc120f640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:35:47.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.715+0000 7fddc120f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:47.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.715+0000 7fddc120f640 1 -- 172.21.15.107:0/225107976 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fddbc139ae0 con 0x7fdda80120a0 2024-03-25T19:35:47.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.917+0000 7fdda37fe640 1 -- 172.21.15.107:0/225107976 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fddb40056f0 con 0x7fdda80120a0 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.917+0000 7fdda37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.917+0000 7fdda37fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 10 librados: watch_flush enter 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 10 librados: watch_flush exit 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 20 client.4112.objecter shutdown clearing up homeless session... 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 10 client.4112.objecter successfully canceled tick 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 10 monclient: shutdown 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 1 -- 172.21.15.107:0/225107976 >> v1:172.21.15.107:6790/0 conn(0x7fdda80120a0 legacy=0x7fddbc1d7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 1 -- 172.21.15.107:0/225107976 shutdown_connections 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 1 -- 172.21.15.107:0/225107976 >> 172.21.15.107:0/225107976 conn(0x7fddbc0b7f40 msgr2=0x7fddbc0b3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:47.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 1 -- 172.21.15.107:0/225107976 shutdown_connections 2024-03-25T19:35:47.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 1 -- 172.21.15.107:0/225107976 wait complete. 2024-03-25T19:35:47.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:47.918+0000 7fddc120f640 1 librados: shutdown 2024-03-25T19:35:47.920 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:47.920+0000 7f18f6e42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:35:47.930 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:35:47.930 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:35:47.930 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:35:47.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:35:47.933 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:47.934+0000 7f45c3e10040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:35:47.959 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:35:47.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:35:47.974 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:47.975+0000 7f18f6e42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:35:47.993 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:47.993+0000 7f45c3e10040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:35:48.019 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:35:48.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:35:48.086 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-25T19:35:48.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:35:48.153 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:35:48.154 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:35:48.180 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:35:48.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:35:48.247 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:35:48.247 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:35:48.314 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-25T19:35:48.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:35:48.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3 0 2024-03-25T19:35:48.515 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:48.515+0000 7f18f6e42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:35:48.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.525+0000 7fbf6f582640 10 monclient: get_monmap_and_config 2024-03-25T19:35:48.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.525+0000 7fbf6f582640 10 monclient: build_initial_monmap 2024-03-25T19:35:48.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.525+0000 7fbf6f582640 10 monclient: monmap: 2024-03-25T19:35:48.524 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:48.525 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:48.525 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:35:48.525954+0000 2024-03-25T19:35:48.525 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:35:48.525954+0000 2024-03-25T19:35:48.525 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:48.525 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:48.525 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:48.525 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:35:48.525 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:35:48.525 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.525+0000 7fbf6f582640 1 Processor -- start 2024-03-25T19:35:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.525+0000 7fbf6f582640 1 -- start start 2024-03-25T19:35:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.525+0000 7fbf6f582640 10 monclient: init 2024-03-25T19:35:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.526+0000 7fbf6f582640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.526+0000 7fbf6f582640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.526+0000 7fbf6f582640 10 monclient(hunting): picked mon.noname-b con 0x7fbf680b4700 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.526+0000 7fbf6f582640 10 monclient(hunting): picked mon.noname-c con 0x7fbf68142a20 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.526+0000 7fbf6f582640 10 monclient(hunting): picked mon.noname-a con 0x7fbf680a96b0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.526+0000 7fbf6f582640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf680a9d60 con 0x7fbf680a96b0 2024-03-25T19:35:48.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.526+0000 7fbf6f582640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6813ff10 con 0x7fbf68142a20 2024-03-25T19:35:48.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.526+0000 7fbf6f582640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf680aa1f0 con 0x7fbf680b4700 2024-03-25T19:35:48.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.526+0000 7fbf6f582640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:48.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.526+0000 7fbf6f582640 10 monclient(hunting): authenticate will time out at 4551.684207s 2024-03-25T19:35:48.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.527+0000 7fbf6cb1d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbf68142a20 0x7fbf6813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52832/0 (socket says 172.21.15.195:52832) 2024-03-25T19:35:48.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.527+0000 7fbf6cb1d640 1 -- 172.21.15.195:0/3924574853 learned_addr learned my addr 172.21.15.195:0/3924574853 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:48.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.529+0000 7fbf5ffff640 1 -- 172.21.15.195:0/3924574853 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109504978 0 0) 0x7fbf6813ff10 con 0x7fbf68142a20 2024-03-25T19:35:48.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.530+0000 7fbf5ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:48.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.530+0000 7fbf5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:48.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.530+0000 7fbf5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:48.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.530+0000 7fbf5ffff640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:35:48.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.530+0000 7fbf5ffff640 1 -- 172.21.15.195:0/3924574853 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf54003520 con 0x7fbf68142a20 2024-03-25T19:35:48.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 1 -- 172.21.15.195:0/3924574853 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 606700745 0 0) 0x7fbf54003520 con 0x7fbf68142a20 2024-03-25T19:35:48.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 1 -- 172.21.15.195:0/3924574853 >> v1:172.21.15.107:6790/0 conn(0x7fbf680b4700 legacy=0x7fbf68142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:35:48.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 1 -- 172.21.15.195:0/3924574853 >> v1:172.21.15.107:6789/0 conn(0x7fbf680a96b0 legacy=0x7fbf6813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:35:48.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 1 monclient: found mon.noname-c 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 1 -- 172.21.15.195:0/3924574853 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf680b4ae0 con 0x7fbf68142a20 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 10 monclient: _finish_auth 0 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 1 -- 172.21.15.195:0/3924574853 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fbf58001690 con 0x7fbf68142a20 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf5ffff640 10 monclient: dump: 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:48.535 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:48.536 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:48.536 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:48.536 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:48.536 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:48.536 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:48.536 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:48.536 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:48.536 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:48.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf6f582640 5 monclient: authenticate success, global_id 4117 2024-03-25T19:35:48.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.534+0000 7fbf6f582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:48.565 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:48.566+0000 7f45c3e10040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:35:48.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.572+0000 7fbf5ffff640 1 -- 172.21.15.195:0/3924574853 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf58001860 con 0x7fbf68142a20 2024-03-25T19:35:48.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.572+0000 7fbf5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:48.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.572+0000 7fbf5ffff640 1 -- 172.21.15.195:0/3924574853 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fbf58001cb0 con 0x7fbf68142a20 2024-03-25T19:35:48.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.572+0000 7fbf5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:48.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.573+0000 7fbf5ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:48.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.573+0000 7fbf5ffff640 10 monclient: dump: 2024-03-25T19:35:48.573 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:48.573 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:48.573 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:48.573 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:48.573 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:48.573 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:48.573 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:48.573 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:48.573 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:48.573 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:48.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.573+0000 7fbf6f582640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:48.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.573+0000 7fbf6f582640 4 set_mon_vals no callback set 2024-03-25T19:35:48.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.573+0000 7fbf6f582640 10 monclient: shutdown 2024-03-25T19:35:48.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.573+0000 7fbf6f582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:48.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.573+0000 7fbf6f582640 1 -- 172.21.15.195:0/3924574853 >> v1:172.21.15.195:6789/0 conn(0x7fbf68142a20 legacy=0x7fbf6813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:48.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.574+0000 7fbf6f582640 1 -- 172.21.15.195:0/3924574853 shutdown_connections 2024-03-25T19:35:48.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.574+0000 7fbf6f582640 1 -- 172.21.15.195:0/3924574853 >> 172.21.15.195:0/3924574853 conn(0x7fbf680668e0 msgr2=0x7fbf68066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:48.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.574+0000 7fbf6f582640 1 -- 172.21.15.195:0/3924574853 shutdown_connections 2024-03-25T19:35:48.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.574+0000 7fbf6f582640 1 -- 172.21.15.195:0/3924574853 wait complete. 2024-03-25T19:35:48.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.574+0000 7fbf6f582640 10 monclient: build_initial_monmap 2024-03-25T19:35:48.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.574+0000 7fbf6f582640 10 monclient: monmap: 2024-03-25T19:35:48.578 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:48.578 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:48.578 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-25T19:35:48.578 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-25T19:35:48.578 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:48.579 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-25T19:35:48.579 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:48.579 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:35:48.579 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:35:48.579 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:48.579 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 1 librados: starting msgr at 2024-03-25T19:35:48.579 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 1 librados: starting objecter 2024-03-25T19:35:48.579 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 1 Processor -- start 2024-03-25T19:35:48.579 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 1 -- start start 2024-03-25T19:35:48.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 1 librados: setting wanted keys 2024-03-25T19:35:48.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 1 librados: calling monclient init 2024-03-25T19:35:48.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 10 monclient: init 2024-03-25T19:35:48.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:48.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:48.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 10 monclient(hunting): picked mon.noname-b con 0x7fbf680a96b0 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:48.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 10 monclient(hunting): picked mon.noname-c con 0x7fbf680b4700 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:48.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 10 monclient(hunting): picked mon.noname-a con 0x7fbf68142a20 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:48.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6813cce0 con 0x7fbf68142a20 2024-03-25T19:35:48.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf681d8310 con 0x7fbf680a96b0 2024-03-25T19:35:48.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf681d76e0 con 0x7fbf680b4700 2024-03-25T19:35:48.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:48.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.575+0000 7fbf6f582640 10 monclient(hunting): authenticate will time out at 4551.733578s 2024-03-25T19:35:48.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.576+0000 7fbf6db1f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbf68142a20 0x7fbf6813c1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.195:44576/0 (socket says 172.21.15.195:44576) 2024-03-25T19:35:48.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.576+0000 7fbf6db1f640 1 -- 172.21.15.195:0/504116615 learned_addr learned my addr 172.21.15.195:0/504116615 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:48.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.576+0000 7fbf5dffb640 10 client.?.objecter ms_handle_connect 0x7fbf68142a20 2024-03-25T19:35:48.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.576+0000 7fbf5dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:48.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.577+0000 7fbf5dffb640 10 client.?.objecter ms_handle_connect 0x7fbf680a96b0 2024-03-25T19:35:48.582 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.577+0000 7fbf5dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:48.582 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.578+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881571722 0 0) 0x7fbf6813cce0 con 0x7fbf68142a20 2024-03-25T19:35:48.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.578+0000 7fbf5dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:48.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.578+0000 7fbf5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:48.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.578+0000 7fbf5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:48.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.578+0000 7fbf5dffb640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:35:48.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.578+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf4c004eb0 con 0x7fbf68142a20 2024-03-25T19:35:48.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.579+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241344986 0 0) 0x7fbf681d8310 con 0x7fbf680a96b0 2024-03-25T19:35:48.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.579+0000 7fbf5dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:48.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.579+0000 7fbf5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:48.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.579+0000 7fbf5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:48.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.579+0000 7fbf5dffb640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:35:48.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.579+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf6813cce0 con 0x7fbf680a96b0 2024-03-25T19:35:48.584 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.583+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4201944757 0 0) 0x7fbf6813cce0 con 0x7fbf680a96b0 2024-03-25T19:35:48.584 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.583+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf681d8310 con 0x7fbf680a96b0 2024-03-25T19:35:48.584 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.583+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fbf64001830 con 0x7fbf680a96b0 2024-03-25T19:35:48.584 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.583+0000 7fbf5dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:48.584 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.583+0000 7fbf5dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:48.584 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.583+0000 7fbf5dffb640 10 monclient(hunting): dump: 2024-03-25T19:35:48.584 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:48.584 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:48.584 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:48.585 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:48.585 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:48.585 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:48.585 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:48.585 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:48.585 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:48.585 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:48.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.583+0000 7fbf5dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:48.587 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:48.588+0000 7f18f6e42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:35:48.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2243953945 0 0) 0x7fbf681d8310 con 0x7fbf680a96b0 2024-03-25T19:35:48.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 >> v1:172.21.15.107:6790/0 conn(0x7fbf680b4700 legacy=0x7fbf6813bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 >> v1:172.21.15.107:6789/0 conn(0x7fbf68142a20 legacy=0x7fbf6813c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf5dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf5dffb640 1 monclient: found mon.b 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf5dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf681d4fe0 con 0x7fbf680a96b0 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf5dffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf5dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf6f582640 5 monclient: authenticate success, global_id 4120 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf6f582640 10 monclient: _renew_subs 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf6f582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:48.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.593+0000 7fbf6f582640 1 -- 172.21.15.195:0/504116615 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf681e2d10 con 0x7fbf680a96b0 2024-03-25T19:35:48.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.594+0000 7fbf6f582640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:48.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.594+0000 7fbf6f582640 10 monclient: _renew_subs 2024-03-25T19:35:48.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.594+0000 7fbf6f582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:48.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.594+0000 7fbf6f582640 1 -- 172.21.15.195:0/504116615 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf681e3080 con 0x7fbf680a96b0 2024-03-25T19:35:48.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.594+0000 7fbf6f582640 1 librados: init done 2024-03-25T19:35:48.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.594+0000 7fbf6f582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:48.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.594+0000 7fbf6f582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:48.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.594+0000 7fbf6f582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:48.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.594+0000 7fbf6f582640 1 -- 172.21.15.195:0/504116615 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf681d7870 con 0x7fbf680a96b0 2024-03-25T19:35:48.634 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:48.635+0000 7f18f6e42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:35:48.643 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:48.643+0000 7f45c3e10040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:35:48.687 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:48.687+0000 7f45c3e10040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:35:48.689 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:48.690+0000 7f18f6e42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:35:48.734 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:48.734+0000 7f18f6e42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:35:48.739 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:48.740+0000 7f45c3e10040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:35:48.780 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:48.781+0000 7f45c3e10040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:35:48.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf64001d40 con 0x7fbf680a96b0 2024-03-25T19:35:48.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:48.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fbf64003790 con 0x7fbf680a96b0 2024-03-25T19:35:48.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:48.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:48.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 10 monclient: dump: 2024-03-25T19:35:48.789 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:48.789 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:48.789 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:48.789 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbf64003c80 con 0x7fbf680a96b0 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5f7fe640 4 set_mon_vals no callback set 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2595400078 0 0) 0x7fbf64005220 con 0x7fbf680a96b0 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 10 client.4120.objecter ms_dispatch 0x7fbf68000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:35:48.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:35:48.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-25T19:35:48.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbf64005580 con 0x7fbf680a96b0 2024-03-25T19:35:48.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:48.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.789+0000 7fbf5dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:48.853 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:48.854+0000 7f18f6e42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:35:48.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.862+0000 7fbf6f582640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3", "id": 0}] 2024-03-25T19:35:48.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.862+0000 7fbf6f582640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3", "id": 0}] 2024-03-25T19:35:48.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.862+0000 7fbf6f582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:48.862 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:48.862+0000 7fbf6f582640 1 -- 172.21.15.195:0/504116615 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3", "id": 0} v 0) v1 -- 0x7fbf6804f730 con 0x7fbf680a96b0 2024-03-25T19:35:48.893 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:48.893+0000 7f45c3e10040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:35:48.924 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:48.925+0000 7f18f6e42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:35:48.969 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:48.970+0000 7f18f6e42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:35:48.984 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:48.984+0000 7f45c3e10040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:35:49.021 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:49.021+0000 7f18f6e42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:35:49.034 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:49.034+0000 7f45c3e10040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:35:49.070 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:49.070+0000 7f18f6e42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:35:49.078 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.079+0000 7fbf5dffb640 1 -- 172.21.15.195:0/504116615 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1848504178 0 732707515) 0x7fbf640057c0 con 0x7fbf680a96b0 2024-03-25T19:35:49.078 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.079+0000 7fbf5dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3", "id": 0}] 2024-03-25T19:35:49.078 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.079+0000 7fbf5dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:49.079 INFO:teuthology.orchestra.run.smithi195.stdout:0 2024-03-25T19:35:49.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.079+0000 7fbf6f582640 10 librados: watch_flush enter 2024-03-25T19:35:49.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.080+0000 7fbf6f582640 10 librados: watch_flush exit 2024-03-25T19:35:49.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.080+0000 7fbf6f582640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-25T19:35:49.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.080+0000 7fbf6f582640 10 client.4120.objecter successfully canceled tick 2024-03-25T19:35:49.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.080+0000 7fbf6f582640 10 monclient: shutdown 2024-03-25T19:35:49.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.080+0000 7fbf6f582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.080+0000 7fbf6f582640 1 -- 172.21.15.195:0/504116615 >> v1:172.21.15.195:6789/0 conn(0x7fbf680a96b0 legacy=0x7fbf681d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.080+0000 7fbf6f582640 1 -- 172.21.15.195:0/504116615 shutdown_connections 2024-03-25T19:35:49.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.080+0000 7fbf6f582640 1 -- 172.21.15.195:0/504116615 >> 172.21.15.195:0/504116615 conn(0x7fbf680668e0 msgr2=0x7fbf680b6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.080+0000 7fbf6f582640 1 -- 172.21.15.195:0/504116615 shutdown_connections 2024-03-25T19:35:49.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.080+0000 7fbf6f582640 1 -- 172.21.15.195:0/504116615 wait complete. 2024-03-25T19:35:49.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.080+0000 7fbf6f582640 1 librados: shutdown 2024-03-25T19:35:49.088 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:49.088+0000 7f45c3e10040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:35:49.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 96605efd-5d37-41af-aae0-6d5ad1736b26 1 2024-03-25T19:35:49.139 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:49.140+0000 7f45c3e10040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:35:49.190 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.190+0000 7f644704e640 10 monclient: get_monmap_and_config 2024-03-25T19:35:49.190 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.190+0000 7f644704e640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.190 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.190+0000 7f644704e640 10 monclient: monmap: 2024-03-25T19:35:49.190 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:49.190 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.190 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:35:49.191498+0000 2024-03-25T19:35:49.190 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:35:49.191498+0000 2024-03-25T19:35:49.191 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.191 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.191 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:49.191 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:35:49.191 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:35:49.191 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.191+0000 7f644704e640 1 Processor -- start 2024-03-25T19:35:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.191+0000 7f644704e640 1 -- start start 2024-03-25T19:35:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.191+0000 7f644704e640 10 monclient: init 2024-03-25T19:35:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.191+0000 7f644704e640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.191+0000 7f644704e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.192+0000 7f644704e640 10 monclient(hunting): picked mon.noname-a con 0x7f64400b4700 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.192+0000 7f644704e640 10 monclient(hunting): picked mon.noname-c con 0x7f6440142a20 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.192+0000 7f644704e640 10 monclient(hunting): picked mon.noname-b con 0x7f64400a96b0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.192+0000 7f644704e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64400a9d60 con 0x7f64400b4700 2024-03-25T19:35:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.192+0000 7f644704e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644013ff10 con 0x7f6440142a20 2024-03-25T19:35:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.192+0000 7f644704e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64400aa1f0 con 0x7f64400a96b0 2024-03-25T19:35:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.192+0000 7f644704e640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.192+0000 7f644704e640 10 monclient(hunting): authenticate will time out at 4552.349867s 2024-03-25T19:35:49.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.192+0000 7f643ffff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6440142a20 0x7f644013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52842/0 (socket says 172.21.15.195:52842) 2024-03-25T19:35:49.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.192+0000 7f643ffff640 1 -- 172.21.15.195:0/1409419883 learned_addr learned my addr 172.21.15.195:0/1409419883 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:49.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526717935 0 0) 0x7f644013ff10 con 0x7f6440142a20 2024-03-25T19:35:49.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:35:49.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6430003520 con 0x7f6440142a20 2024-03-25T19:35:49.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025405865 0 0) 0x7f64400aa1f0 con 0x7f64400a96b0 2024-03-25T19:35:49.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:35:49.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f644013ff10 con 0x7f64400a96b0 2024-03-25T19:35:49.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439419697 0 0) 0x7f64400a9d60 con 0x7f64400b4700 2024-03-25T19:35:49.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:35:49.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.195+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64400aa1f0 con 0x7f64400b4700 2024-03-25T19:35:49.198 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2812340382 0 0) 0x7f644013ff10 con 0x7f64400a96b0 2024-03-25T19:35:49.198 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 >> v1:172.21.15.195:6789/0 conn(0x7f6440142a20 legacy=0x7f644013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.198 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 >> v1:172.21.15.107:6789/0 conn(0x7f64400b4700 legacy=0x7f6440142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.198 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.198 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 1 monclient: found mon.noname-b 2024-03-25T19:35:49.198 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.198 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:35:49.198 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64400b4ae0 con 0x7f64400a96b0 2024-03-25T19:35:49.199 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.199 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.199 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.199 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f64380019f0 con 0x7f64400a96b0 2024-03-25T19:35:49.199 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.199 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:49.199 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f643f7fe640 10 monclient: dump: 2024-03-25T19:35:49.199 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.199 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.200 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.200 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.200 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.200 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.200 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.200 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.200 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.200 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.200 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f644704e640 5 monclient: authenticate success, global_id 4118 2024-03-25T19:35:49.200 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.198+0000 7f644704e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:49.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6438001b50 con 0x7f64400a96b0 2024-03-25T19:35:49.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f643f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f643f7fe640 1 -- 172.21.15.195:0/1409419883 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6438001f60 con 0x7f64400a96b0 2024-03-25T19:35:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f644704e640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f644704e640 4 set_mon_vals no callback set 2024-03-25T19:35:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f643f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f643f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f643f7fe640 10 monclient: dump: 2024-03-25T19:35:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.204 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f644704e640 10 monclient: shutdown 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f644704e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f644704e640 1 -- 172.21.15.195:0/1409419883 >> v1:172.21.15.107:6790/0 conn(0x7f64400a96b0 legacy=0x7f644013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f644704e640 1 -- 172.21.15.195:0/1409419883 shutdown_connections 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f644704e640 1 -- 172.21.15.195:0/1409419883 >> 172.21.15.195:0/1409419883 conn(0x7f64400668e0 msgr2=0x7f6440066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f644704e640 1 -- 172.21.15.195:0/1409419883 shutdown_connections 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.200+0000 7f644704e640 1 -- 172.21.15.195:0/1409419883 wait complete. 2024-03-25T19:35:49.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.201+0000 7f644704e640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.201+0000 7f644704e640 10 monclient: monmap: 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.201+0000 7f644704e640 1 librados: starting msgr at 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.201+0000 7f644704e640 1 librados: starting objecter 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.201+0000 7f644704e640 1 Processor -- start 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 1 -- start start 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 1 librados: setting wanted keys 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 1 librados: calling monclient init 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 10 monclient: init 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 10 monclient(hunting): picked mon.noname-a con 0x7f64400a96b0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 10 monclient(hunting): picked mon.noname-b con 0x7f64400b4700 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 10 monclient(hunting): picked mon.noname-c con 0x7f6440142a20 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64401349a0 con 0x7f64400a96b0 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64400aba20 con 0x7f64400b4700 2024-03-25T19:35:49.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644014e6c0 con 0x7f6440142a20 2024-03-25T19:35:49.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.202+0000 7f644704e640 10 monclient(hunting): authenticate will time out at 4552.360514s 2024-03-25T19:35:49.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.203+0000 7f64455eb640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6440142a20 0x7f6440134290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.195:33516/0 (socket says 172.21.15.195:33516) 2024-03-25T19:35:49.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.203+0000 7f64455eb640 1 -- 172.21.15.195:0/1149945436 learned_addr learned my addr 172.21.15.195:0/1149945436 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:49.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.203+0000 7f643d7fa640 10 client.?.objecter ms_handle_connect 0x7f6440142a20 2024-03-25T19:35:49.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.203+0000 7f643d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.203+0000 7f643d7fa640 10 client.?.objecter ms_handle_connect 0x7f64400a96b0 2024-03-25T19:35:49.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.203+0000 7f643d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.205+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824692857 0 0) 0x7f644014e6c0 con 0x7f6440142a20 2024-03-25T19:35:49.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.205+0000 7f643d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.205+0000 7f643d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.205+0000 7f643d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.205+0000 7f643d7fa640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:35:49.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.205+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6424004eb0 con 0x7f6440142a20 2024-03-25T19:35:49.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.205+0000 7f643d7fa640 10 client.?.objecter ms_handle_connect 0x7f64400b4700 2024-03-25T19:35:49.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.205+0000 7f643d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.206+0000 7f6444dea640 1 -- 172.21.15.195:0/1149945436 >> v1:172.21.15.107:6789/0 conn(0x7f64400a96b0 legacy=0x7f64401d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:35:49.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.206+0000 7f6444dea640 1 -- 172.21.15.195:0/1149945436 >> v1:172.21.15.107:6789/0 conn(0x7f64400a96b0 legacy=0x7f64401d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:35:49.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.206+0000 7f6444dea640 1 --1- 172.21.15.195:0/1149945436 >> v1:172.21.15.107:6789/0 conn(0x7f64400a96b0 0x7f64401d7560 :-1 s=OPENED pgs=20 cs=1 l=1).handle_message read tag failed 2024-03-25T19:35:49.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.206+0000 7f6444dea640 1 --1- 172.21.15.195:0/1149945436 >> v1:172.21.15.107:6789/0 conn(0x7f64400a96b0 0x7f64401d7560 :-1 s=OPENED pgs=20 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:35:49.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.206+0000 7f643d7fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.107:6789/0 2024-03-25T19:35:49.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.207+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20432798 0 0) 0x7f64400aba20 con 0x7f64400b4700 2024-03-25T19:35:49.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.207+0000 7f643d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.207+0000 7f643d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.207+0000 7f643d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.207+0000 7f643d7fa640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:35:49.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.207+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f644014e6c0 con 0x7f64400b4700 2024-03-25T19:35:49.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.210+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1976374867 0 0) 0x7f6424004eb0 con 0x7f6440142a20 2024-03-25T19:35:49.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.210+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64400aba20 con 0x7f6440142a20 2024-03-25T19:35:49.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.210+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6438001fa0 con 0x7f6440142a20 2024-03-25T19:35:49.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.210+0000 7f643d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.210+0000 7f643d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:49.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.210+0000 7f643d7fa640 10 monclient(hunting): dump: 2024-03-25T19:35:49.212 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.212 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.212 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.212 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.213 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.213 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.213 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.213 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.213 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.213 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.213 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.210+0000 7f643d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:49.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.212+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1097158446 0 0) 0x7f644014e6c0 con 0x7f64400b4700 2024-03-25T19:35:49.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.212+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6424004eb0 con 0x7f64400b4700 2024-03-25T19:35:49.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.212+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f642c001950 con 0x7f64400b4700 2024-03-25T19:35:49.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.212+0000 7f643d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.212+0000 7f643d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:49.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.212+0000 7f643d7fa640 10 monclient(hunting): dump: 2024-03-25T19:35:49.215 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.215 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.215 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3341555727 0 0) 0x7f64400aba20 con 0x7f6440142a20 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 >> v1:172.21.15.195:6789/0 conn(0x7f64400b4700 legacy=0x7f6440133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 >> v1:172.21.15.107:6789/0 conn(0x7f64400a96b0 legacy=0x7f64401d7560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f643d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f643d7fa640 1 monclient: found mon.c 2024-03-25T19:35:49.216 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f643d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f643d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644014e850 con 0x7f6440142a20 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f643d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f643d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f643d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f644704e640 5 monclient: authenticate success, global_id 4121 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f644704e640 10 monclient: _renew_subs 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f644704e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f644704e640 1 -- 172.21.15.195:0/1149945436 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644013be90 con 0x7f6440142a20 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f644704e640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f644704e640 10 monclient: _renew_subs 2024-03-25T19:35:49.217 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f644704e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:49.218 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f644704e640 1 -- 172.21.15.195:0/1149945436 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f644013ab50 con 0x7f6440142a20 2024-03-25T19:35:49.218 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.213+0000 7f644704e640 1 librados: init done 2024-03-25T19:35:49.218 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.214+0000 7f644704e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.218 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.214+0000 7f644704e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.218 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.214+0000 7f644704e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:49.218 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.214+0000 7f644704e640 1 -- 172.21.15.195:0/1149945436 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644004f730 con 0x7f6440142a20 2024-03-25T19:35:49.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6438003c20 con 0x7f6440142a20 2024-03-25T19:35:49.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6438003dd0 con 0x7f6440142a20 2024-03-25T19:35:49.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:49.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 10 monclient: dump: 2024-03-25T19:35:49.219 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643e7fc640 4 set_mon_vals no callback set 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6438004380 con 0x7f6440142a20 2024-03-25T19:35:49.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1826987163 0 0) 0x7f6438006860 con 0x7f6440142a20 2024-03-25T19:35:49.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 10 client.4121.objecter ms_dispatch 0x7f6440000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:35:49.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 3 client.4121.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:35:49.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 3 client.4121.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:35:49.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.217+0000 7f643d7fa640 20 client.4121.objecter dump_active .. 0 homeless 2024-03-25T19:35:49.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.312+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6438004630 con 0x7f6440142a20 2024-03-25T19:35:49.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.312+0000 7f643d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.312+0000 7f643d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:49.384 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.384+0000 7f644704e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "96605efd-5d37-41af-aae0-6d5ad1736b26", "id": 1}] 2024-03-25T19:35:49.384 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.384+0000 7f644704e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "96605efd-5d37-41af-aae0-6d5ad1736b26", "id": 1}] 2024-03-25T19:35:49.385 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.384+0000 7f644704e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:49.385 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.384+0000 7f644704e640 1 -- 172.21.15.195:0/1149945436 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd new", "uuid": "96605efd-5d37-41af-aae0-6d5ad1736b26", "id": 1} v 0) v1 -- 0x7f6440069f30 con 0x7f6440142a20 2024-03-25T19:35:49.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.442+0000 7f643d7fa640 1 -- 172.21.15.195:0/1149945436 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "96605efd-5d37-41af-aae0-6d5ad1736b26", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3178382370 0 940483276) 0x7f64380047d0 con 0x7f6440142a20 2024-03-25T19:35:49.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.442+0000 7f643d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "96605efd-5d37-41af-aae0-6d5ad1736b26", "id": 1}] 2024-03-25T19:35:49.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.442+0000 7f643d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:49.442 INFO:teuthology.orchestra.run.smithi195.stdout:1 2024-03-25T19:35:49.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 10 librados: watch_flush enter 2024-03-25T19:35:49.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 10 librados: watch_flush exit 2024-03-25T19:35:49.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 20 client.4121.objecter shutdown clearing up homeless session... 2024-03-25T19:35:49.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 10 client.4121.objecter successfully canceled tick 2024-03-25T19:35:49.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 10 monclient: shutdown 2024-03-25T19:35:49.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 1 -- 172.21.15.195:0/1149945436 >> v1:172.21.15.107:6790/0 conn(0x7f6440142a20 legacy=0x7f6440134290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 1 -- 172.21.15.195:0/1149945436 shutdown_connections 2024-03-25T19:35:49.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 1 -- 172.21.15.195:0/1149945436 >> 172.21.15.195:0/1149945436 conn(0x7f64400668e0 msgr2=0x7f644013d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 1 -- 172.21.15.195:0/1149945436 shutdown_connections 2024-03-25T19:35:49.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 1 -- 172.21.15.195:0/1149945436 wait complete. 2024-03-25T19:35:49.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.443+0000 7f644704e640 1 librados: shutdown 2024-03-25T19:35:49.453 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 6d185d22-f8a7-4be4-b82c-19da029c2520 2 2024-03-25T19:35:49.464 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:49.465+0000 7f18f6e42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:35:49.535 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:49.535+0000 7f45c3e10040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:35:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.554+0000 7f953e3b5640 10 monclient: get_monmap_and_config 2024-03-25T19:35:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.554+0000 7f953e3b5640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.554+0000 7f953e3b5640 10 monclient: monmap: 2024-03-25T19:35:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:35:49.555091+0000 2024-03-25T19:35:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:35:49.555091+0000 2024-03-25T19:35:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:49.593 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.554+0000 7f953e3b5640 1 Processor -- start 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 1 -- start start 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 10 monclient: init 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 10 monclient(hunting): picked mon.noname-b con 0x7f95380b4700 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 10 monclient(hunting): picked mon.noname-c con 0x7f9538142a20 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 10 monclient(hunting): picked mon.noname-a con 0x7f95380a96b0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95380a9d60 con 0x7f95380a96b0 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f953813ff10 con 0x7f9538142a20 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95380aa1f0 con 0x7f95380b4700 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.555+0000 7f953e3b5640 10 monclient(hunting): authenticate will time out at 4552.713449s 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.556+0000 7f95377fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9538142a20 0x7f953813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52860/0 (socket says 172.21.15.195:52860) 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.556+0000 7f95377fe640 1 -- 172.21.15.195:0/671043773 learned_addr learned my addr 172.21.15.195:0/671043773 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049090959 0 0) 0x7f95380aa1f0 con 0x7f95380b4700 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9520003520 con 0x7f95380b4700 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539721444 0 0) 0x7f953813ff10 con 0x7f9538142a20 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.558+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95380aa1f0 con 0x7f9538142a20 2024-03-25T19:35:49.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.559+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515444055 0 0) 0x7f95380a9d60 con 0x7f95380a96b0 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.559+0000 7f9536ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.559+0000 7f9536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.559+0000 7f9536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.559+0000 7f9536ffd640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.559+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f953813ff10 con 0x7f95380a96b0 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.561+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2710198385 0 0) 0x7f95380aa1f0 con 0x7f9538142a20 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 >> v1:172.21.15.107:6790/0 conn(0x7f95380b4700 legacy=0x7f9538142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 >> v1:172.21.15.107:6789/0 conn(0x7f95380a96b0 legacy=0x7f953813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 1 monclient: found mon.noname-c 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95380b4ae0 con 0x7f9538142a20 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f953e3b5640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f953e3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f952c001630 con 0x7f9538142a20 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f9536ffd640 10 monclient: dump: 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.562+0000 7f953e3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:49.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.563+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f952c001880 con 0x7f9538142a20 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.563+0000 7f9536ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.563+0000 7f953e3b5640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.563+0000 7f953e3b5640 4 set_mon_vals no callback set 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.563+0000 7f9536ffd640 1 -- 172.21.15.195:0/671043773 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f952c001c10 con 0x7f9538142a20 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.563+0000 7f9536ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.563+0000 7f9536ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.563+0000 7f9536ffd640 10 monclient: dump: 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.598 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.564+0000 7f953e3b5640 10 monclient: shutdown 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.564+0000 7f953e3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.564+0000 7f953e3b5640 1 -- 172.21.15.195:0/671043773 >> v1:172.21.15.195:6789/0 conn(0x7f9538142a20 legacy=0x7f953813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.564+0000 7f953e3b5640 1 -- 172.21.15.195:0/671043773 shutdown_connections 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.564+0000 7f953e3b5640 1 -- 172.21.15.195:0/671043773 >> 172.21.15.195:0/671043773 conn(0x7f95380668e0 msgr2=0x7f9538066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.564+0000 7f953e3b5640 1 -- 172.21.15.195:0/671043773 shutdown_connections 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.564+0000 7f953e3b5640 1 -- 172.21.15.195:0/671043773 wait complete. 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.565+0000 7f953e3b5640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.565+0000 7f953e3b5640 10 monclient: monmap: 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-25T19:35:49.599 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.565+0000 7f953e3b5640 1 librados: starting msgr at 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.565+0000 7f953e3b5640 1 librados: starting objecter 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.565+0000 7f953e3b5640 1 Processor -- start 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.565+0000 7f953e3b5640 1 -- start start 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.565+0000 7f953e3b5640 1 librados: setting wanted keys 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.565+0000 7f953e3b5640 1 librados: calling monclient init 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.565+0000 7f953e3b5640 10 monclient: init 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f953e3b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f953e3b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f953e3b5640 10 monclient(hunting): picked mon.noname-c con 0x7f95380a96b0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f953e3b5640 10 monclient(hunting): picked mon.noname-b con 0x7f95380b4700 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:49.600 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f953e3b5640 10 monclient(hunting): picked mon.noname-a con 0x7f9538142a20 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f953e3b5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f953813c260 con 0x7f9538142a20 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f953e3b5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95380abd70 con 0x7f95380b4700 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f953e3b5640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95380abfe0 con 0x7f95380a96b0 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f953e3b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f953e3b5640 10 monclient(hunting): authenticate will time out at 4552.724102s 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f95377fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f95380b4700 0x7f95380ac400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52864/0 (socket says 172.21.15.195:52864) 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.566+0000 7f95377fe640 1 -- 172.21.15.195:0/1559419363 learned_addr learned my addr 172.21.15.195:0/1559419363 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.567+0000 7f9534ff9640 10 client.?.objecter ms_handle_connect 0x7f95380b4700 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.567+0000 7f9534ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.567+0000 7f9534ff9640 10 client.?.objecter ms_handle_connect 0x7f9538142a20 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.567+0000 7f9534ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.568+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237695425 0 0) 0x7f95380abd70 con 0x7f95380b4700 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.568+0000 7f9534ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.568+0000 7f9534ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.568+0000 7f9534ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.568+0000 7f9534ff9640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:35:49.601 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.568+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9524004eb0 con 0x7f95380b4700 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.569+0000 7f9534ff9640 10 client.?.objecter ms_handle_connect 0x7f95380a96b0 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.569+0000 7f9534ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.569+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104733337 0 0) 0x7f953813c260 con 0x7f9538142a20 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.569+0000 7f9534ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.569+0000 7f9534ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.569+0000 7f9534ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.569+0000 7f9534ff9640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.569+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95380abd70 con 0x7f9538142a20 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.571+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585634116 0 0) 0x7f95380abfe0 con 0x7f95380a96b0 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.571+0000 7f9534ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.571+0000 7f9534ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.571+0000 7f9534ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.571+0000 7f9534ff9640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.571+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f953813c260 con 0x7f95380a96b0 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.573+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 924851128 0 0) 0x7f9524004eb0 con 0x7f95380b4700 2024-03-25T19:35:49.602 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.573+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95380abfe0 con 0x7f95380b4700 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.573+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f952c001c50 con 0x7f95380b4700 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.573+0000 7f9534ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.573+0000 7f9534ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.573+0000 7f9534ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.573+0000 7f9534ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.574+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1765950985 0 0) 0x7f95380abd70 con 0x7f9538142a20 2024-03-25T19:35:49.603 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.575+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9524004eb0 con 0x7f9538142a20 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.575+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f9530001bf0 con 0x7f9538142a20 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.575+0000 7f9534ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.575+0000 7f9534ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.575+0000 7f9534ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.575+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4182093824 0 0) 0x7f953813c260 con 0x7f95380a96b0 2024-03-25T19:35:49.604 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.575+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95380abd70 con 0x7f95380a96b0 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.575+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f95280017b0 con 0x7f95380a96b0 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.575+0000 7f9534ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.575+0000 7f9534ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.575+0000 7f9534ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1738083162 0 0) 0x7f95380abfe0 con 0x7f95380b4700 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 >> v1:172.21.15.107:6790/0 conn(0x7f95380a96b0 legacy=0x7f9538139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.605 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 >> v1:172.21.15.107:6789/0 conn(0x7f9538142a20 legacy=0x7f95381e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f9534ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f9534ff9640 1 monclient: found mon.b 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f9534ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f9534ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953813cca0 con 0x7f95380b4700 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f9534ff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f9534ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f9534ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f953e3b5640 5 monclient: authenticate success, global_id 4132 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f953e3b5640 10 monclient: _renew_subs 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f953e3b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f953e3b5640 1 -- 172.21.15.195:0/1559419363 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f953813aee0 con 0x7f95380b4700 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f953e3b5640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f953e3b5640 10 monclient: _renew_subs 2024-03-25T19:35:49.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f953e3b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f953e3b5640 1 -- 172.21.15.195:0/1559419363 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f953813bc40 con 0x7f95380b4700 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.576+0000 7f953e3b5640 1 librados: init done 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.577+0000 7f953e3b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.577+0000 7f953e3b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.577+0000 7f953e3b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.577+0000 7f953e3b5640 1 -- 172.21.15.195:0/1559419363 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95380aab00 con 0x7f95380b4700 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f952c0011f0 con 0x7f95380b4700 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f952c004c30 con 0x7f95380b4700 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 10 monclient: dump: 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.607 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f952c0051a0 con 0x7f95380b4700 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1605556421 0 0) 0x7f952c0067f0 con 0x7f95380b4700 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 10 client.4132.objecter ms_dispatch 0x7f9538000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f95367fc640 4 set_mon_vals no callback set 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:35:49.608 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.580+0000 7f9534ff9640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T19:35:49.608 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:49.607+0000 7f18f6e42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:35:49.673 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:49.674+0000 7f45c3e10040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:35:49.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.678+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f952c0064e0 con 0x7f95380b4700 2024-03-25T19:35:49.678 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.678+0000 7f9534ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.678 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.678+0000 7f9534ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:49.726 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:49.726+0000 7f18f6e42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:35:49.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.749+0000 7f953e3b5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6d185d22-f8a7-4be4-b82c-19da029c2520", "id": 2}] 2024-03-25T19:35:49.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.749+0000 7f953e3b5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6d185d22-f8a7-4be4-b82c-19da029c2520", "id": 2}] 2024-03-25T19:35:49.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.749+0000 7f953e3b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:49.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.749+0000 7f953e3b5640 1 -- 172.21.15.195:0/1559419363 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6d185d22-f8a7-4be4-b82c-19da029c2520", "id": 2} v 0) v1 -- 0x7f953804f730 con 0x7f95380b4700 2024-03-25T19:35:49.772 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:49.772+0000 7f18f6e42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:35:49.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.809+0000 7f9534ff9640 1 -- 172.21.15.195:0/1559419363 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6d185d22-f8a7-4be4-b82c-19da029c2520", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3739828238 0 216599125) 0x7f952c006a00 con 0x7f95380b4700 2024-03-25T19:35:49.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.809+0000 7f9534ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6d185d22-f8a7-4be4-b82c-19da029c2520", "id": 2}] 2024-03-25T19:35:49.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.809+0000 7f9534ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:49.809 INFO:teuthology.orchestra.run.smithi195.stdout:2 2024-03-25T19:35:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 10 librados: watch_flush enter 2024-03-25T19:35:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 10 librados: watch_flush exit 2024-03-25T19:35:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T19:35:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 10 client.4132.objecter successfully canceled tick 2024-03-25T19:35:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 10 monclient: shutdown 2024-03-25T19:35:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 1 -- 172.21.15.195:0/1559419363 >> v1:172.21.15.195:6789/0 conn(0x7f95380b4700 legacy=0x7f95380ac400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 1 -- 172.21.15.195:0/1559419363 shutdown_connections 2024-03-25T19:35:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 1 -- 172.21.15.195:0/1559419363 >> 172.21.15.195:0/1559419363 conn(0x7f95380668e0 msgr2=0x7f95380b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 1 -- 172.21.15.195:0/1559419363 shutdown_connections 2024-03-25T19:35:49.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 1 -- 172.21.15.195:0/1559419363 wait complete. 2024-03-25T19:35:49.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.810+0000 7f953e3b5640 1 librados: shutdown 2024-03-25T19:35:49.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2 3 2024-03-25T19:35:49.889 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:49.890+0000 7f18f6e42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:35:49.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.960+0000 7fb2c920f640 10 monclient: get_monmap_and_config 2024-03-25T19:35:49.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.960+0000 7fb2c920f640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.960+0000 7fb2c920f640 10 monclient: monmap: 2024-03-25T19:35:49.961 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:49.961 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.962 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:35:49.961752+0000 2024-03-25T19:35:49.962 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:35:49.961752+0000 2024-03-25T19:35:49.962 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.962 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.962 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:49.962 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:35:49.963 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:35:49.963 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.961+0000 7fb2c920f640 1 Processor -- start 2024-03-25T19:35:49.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.961+0000 7fb2c920f640 1 -- start start 2024-03-25T19:35:49.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.961+0000 7fb2c920f640 10 monclient: init 2024-03-25T19:35:49.966 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c920f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.966 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c920f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:49.966 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c920f640 10 monclient(hunting): picked mon.noname-c con 0x7fb2c40b4700 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:49.966 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c920f640 10 monclient(hunting): picked mon.noname-b con 0x7fb2c4142a20 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:49.966 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c920f640 10 monclient(hunting): picked mon.noname-a con 0x7fb2c40a96b0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:49.966 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c920f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c40a9d60 con 0x7fb2c40a96b0 2024-03-25T19:35:49.966 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c920f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c413ff10 con 0x7fb2c40b4700 2024-03-25T19:35:49.966 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c920f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c40aa1f0 con 0x7fb2c4142a20 2024-03-25T19:35:49.966 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c920f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c920f640 10 monclient(hunting): authenticate will time out at 4553.119863s 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c2d9d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb2c40b4700 0x7fb2c4142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52270/0 (socket says 172.21.15.195:52270) 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.962+0000 7fb2c2d9d640 1 -- 172.21.15.195:0/1779730050 learned_addr learned my addr 172.21.15.195:0/1779730050 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222438654 0 0) 0x7fb2c413ff10 con 0x7fb2c40b4700 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2ac003520 con 0x7fb2c40b4700 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889923232 0 0) 0x7fb2c40a9d60 con 0x7fb2c40a96b0 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:35:49.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c413ff10 con 0x7fb2c40a96b0 2024-03-25T19:35:49.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990938849 0 0) 0x7fb2c40aa1f0 con 0x7fb2c4142a20 2024-03-25T19:35:49.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:35:49.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.965+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c40a9d60 con 0x7fb2c4142a20 2024-03-25T19:35:49.969 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3665522034 0 0) 0x7fb2ac003520 con 0x7fb2c40b4700 2024-03-25T19:35:49.969 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 >> v1:172.21.15.107:6790/0 conn(0x7fb2c4142a20 legacy=0x7fb2c413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 >> v1:172.21.15.107:6789/0 conn(0x7fb2c40a96b0 legacy=0x7fb2c413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 1 monclient: found mon.noname-c 2024-03-25T19:35:49.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:35:49.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40b4ae0 con 0x7fb2c40b4700 2024-03-25T19:35:49.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb2b8001690 con 0x7fb2c40b4700 2024-03-25T19:35:49.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c920f640 5 monclient: authenticate success, global_id 4135 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c920f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c1d9b640 10 monclient: dump: 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.971 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.968+0000 7fb2c920f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2b80017f0 con 0x7fb2c40b4700 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c1d9b640 1 -- 172.21.15.195:0/1779730050 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb2b8001c00 con 0x7fb2c40b4700 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c920f640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c920f640 4 set_mon_vals no callback set 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c1d9b640 10 monclient: dump: 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.972 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.973 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.973 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.973 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.973 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.973 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.973 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.973 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.976 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c920f640 10 monclient: shutdown 2024-03-25T19:35:49.976 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c920f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.976 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c920f640 1 -- 172.21.15.195:0/1779730050 >> v1:172.21.15.195:6789/0 conn(0x7fb2c40b4700 legacy=0x7fb2c4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.976 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c920f640 1 -- 172.21.15.195:0/1779730050 shutdown_connections 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c920f640 1 -- 172.21.15.195:0/1779730050 >> 172.21.15.195:0/1779730050 conn(0x7fb2c40668e0 msgr2=0x7fb2c4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.970+0000 7fb2c920f640 1 -- 172.21.15.195:0/1779730050 shutdown_connections 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.971+0000 7fb2c920f640 1 -- 172.21.15.195:0/1779730050 wait complete. 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.971+0000 7fb2c920f640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.971+0000 7fb2c920f640 10 monclient: monmap: 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-25T19:35:49.977 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:49.978 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:35:49.978 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:35:49.978 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 1 librados: starting msgr at 2024-03-25T19:35:49.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 1 librados: starting objecter 2024-03-25T19:35:49.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 1 Processor -- start 2024-03-25T19:35:49.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 1 -- start start 2024-03-25T19:35:49.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 1 librados: setting wanted keys 2024-03-25T19:35:49.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 1 librados: calling monclient init 2024-03-25T19:35:49.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 10 monclient: init 2024-03-25T19:35:49.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:49.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 10 monclient(hunting): picked mon.noname-a con 0x7fb2c40a96b0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:49.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 10 monclient(hunting): picked mon.noname-c con 0x7fb2c40b4700 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:49.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 10 monclient(hunting): picked mon.noname-b con 0x7fb2c4142a20 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:49.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c413a960 con 0x7fb2c40a96b0 2024-03-25T19:35:49.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c4139340 con 0x7fb2c4142a20 2024-03-25T19:35:49.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c4138db0 con 0x7fb2c40b4700 2024-03-25T19:35:49.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.972+0000 7fb2c920f640 10 monclient(hunting): authenticate will time out at 4553.130708s 2024-03-25T19:35:49.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.973+0000 7fb2c359e640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb2c4142a20 0x7fb2c41daf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52280/0 (socket says 172.21.15.195:52280) 2024-03-25T19:35:49.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.973+0000 7fb2c359e640 1 -- 172.21.15.195:0/1205157820 learned_addr learned my addr 172.21.15.195:0/1205157820 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:49.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.973+0000 7fb2ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb2c4142a20 2024-03-25T19:35:49.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.973+0000 7fb2ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.973+0000 7fb2ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb2c40b4700 2024-03-25T19:35:49.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.973+0000 7fb2ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.973+0000 7fb2ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb2c40a96b0 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.973+0000 7fb2ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.975+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602271852 0 0) 0x7fb2c4138db0 con 0x7fb2c40b4700 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.975+0000 7fb2ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.975+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.975+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.975+0000 7fb2ab7fe640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.975+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b0004eb0 con 0x7fb2c40b4700 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96301916 0 0) 0x7fb2c413a960 con 0x7fb2c40a96b0 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c4138db0 con 0x7fb2c40a96b0 2024-03-25T19:35:49.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597449199 0 0) 0x7fb2c4139340 con 0x7fb2c4142a20 2024-03-25T19:35:49.982 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.982 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.982 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.982 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:35:49.982 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.976+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c413a960 con 0x7fb2c4142a20 2024-03-25T19:35:49.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2944349223 0 0) 0x7fb2b0004eb0 con 0x7fb2c40b4700 2024-03-25T19:35:49.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2c4139340 con 0x7fb2c40b4700 2024-03-25T19:35:49.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb2b40017f0 con 0x7fb2c40b4700 2024-03-25T19:35:49.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:49.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 10 monclient(hunting): dump: 2024-03-25T19:35:49.985 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.985 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.985 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3981629574 0 0) 0x7fb2c413a960 con 0x7fb2c4142a20 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2b0004eb0 con 0x7fb2c4142a20 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb2bc001c30 con 0x7fb2c4142a20 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.986 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:49.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 10 monclient(hunting): dump: 2024-03-25T19:35:49.987 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.987 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.987 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.987 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.987 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.987 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.987 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.987 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.987 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.987 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2978062046 0 0) 0x7fb2c4138db0 con 0x7fb2c40a96b0 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2c413a960 con 0x7fb2c40a96b0 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb2b8001b00 con 0x7fb2c40a96b0 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.980+0000 7fb2ab7fe640 10 monclient(hunting): dump: 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.988 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.989 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.989 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.989 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.989 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 655017593 0 0) 0x7fb2c4139340 con 0x7fb2c40b4700 2024-03-25T19:35:49.989 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 >> v1:172.21.15.195:6789/0 conn(0x7fb2c4142a20 legacy=0x7fb2c41daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.989 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 >> v1:172.21.15.107:6789/0 conn(0x7fb2c40a96b0 legacy=0x7fb2c4130f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.989 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.989 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2ab7fe640 1 monclient: found mon.c 2024-03-25T19:35:49.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:49.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c4138f40 con 0x7fb2c40b4700 2024-03-25T19:35:49.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2ab7fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2c920f640 5 monclient: authenticate success, global_id 4133 2024-03-25T19:35:49.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2c920f640 10 monclient: _renew_subs 2024-03-25T19:35:49.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2c920f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:49.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2c920f640 1 -- 172.21.15.195:0/1205157820 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c4138750 con 0x7fb2c40b4700 2024-03-25T19:35:49.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2c920f640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:49.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2c920f640 10 monclient: _renew_subs 2024-03-25T19:35:49.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2c920f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:49.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2c920f640 1 -- 172.21.15.195:0/1205157820 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c4131640 con 0x7fb2c40b4700 2024-03-25T19:35:49.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.983+0000 7fb2c920f640 1 librados: init done 2024-03-25T19:35:49.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.985+0000 7fb2c920f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.985+0000 7fb2c920f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.985+0000 7fb2c920f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:49.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.985+0000 7fb2c920f640 1 -- 172.21.15.195:0/1205157820 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c4139190 con 0x7fb2c40b4700 2024-03-25T19:35:49.993 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2b4001ce0 con 0x7fb2c40b4700 2024-03-25T19:35:49.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb2b40037b0 con 0x7fb2c40b4700 2024-03-25T19:35:49.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:49.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 10 monclient: dump: 2024-03-25T19:35:49.994 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:49.994 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:49.994 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.994 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:49.994 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb2b4003ca0 con 0x7fb2c40b4700 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2c159a640 4 set_mon_vals no callback set 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2507686665 0 0) 0x7fb2b4005330 con 0x7fb2c40b4700 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 10 client.4133.objecter ms_dispatch 0x7fb2c4000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 3 client.4133.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 3 client.4133.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:35:49.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:49.987+0000 7fb2ab7fe640 20 client.4133.objecter dump_active .. 0 homeless 2024-03-25T19:35:49.996 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:49.992+0000 7f18f6e42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:35:50.059 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:50.059+0000 7f45c3e10040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:35:50.078 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.078+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb2b4004fb0 con 0x7fb2c40b4700 2024-03-25T19:35:50.078 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.078+0000 7fb2ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:50.078 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.078+0000 7fb2ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:50.105 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:50.105+0000 7f45c3e10040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:35:50.149 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:50.149+0000 7f18f6e42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:35:50.150 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.151+0000 7fb2c920f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2", "id": 3}] 2024-03-25T19:35:50.151 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.151+0000 7fb2c920f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2", "id": 3}] 2024-03-25T19:35:50.151 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.151+0000 7fb2c920f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:50.151 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.151+0000 7fb2c920f640 1 -- 172.21.15.195:0/1205157820 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd new", "uuid": "3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2", "id": 3} v 0) v1 -- 0x7fb2c41394d0 con 0x7fb2c40b4700 2024-03-25T19:35:50.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.192+0000 7fb2ab7fe640 1 -- 172.21.15.195:0/1205157820 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3939668335 0 525046306) 0x7fb2b4005540 con 0x7fb2c40b4700 2024-03-25T19:35:50.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.192+0000 7fb2ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2", "id": 3}] 2024-03-25T19:35:50.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.192+0000 7fb2ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:50.192 INFO:teuthology.orchestra.run.smithi195.stdout:3 2024-03-25T19:35:50.193 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:50.193+0000 7f18f6e42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:35:50.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 10 librados: watch_flush enter 2024-03-25T19:35:50.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 10 librados: watch_flush exit 2024-03-25T19:35:50.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 20 client.4133.objecter shutdown clearing up homeless session... 2024-03-25T19:35:50.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 10 client.4133.objecter successfully canceled tick 2024-03-25T19:35:50.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 10 monclient: shutdown 2024-03-25T19:35:50.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:50.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 1 -- 172.21.15.195:0/1205157820 >> v1:172.21.15.107:6790/0 conn(0x7fb2c40b4700 legacy=0x7fb2c41da7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 1 -- 172.21.15.195:0/1205157820 shutdown_connections 2024-03-25T19:35:50.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 1 -- 172.21.15.195:0/1205157820 >> 172.21.15.195:0/1205157820 conn(0x7fb2c40668e0 msgr2=0x7fb2c40b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:50.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 1 -- 172.21.15.195:0/1205157820 shutdown_connections 2024-03-25T19:35:50.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 1 -- 172.21.15.195:0/1205157820 wait complete. 2024-03-25T19:35:50.195 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.193+0000 7fb2c920f640 1 librados: shutdown 2024-03-25T19:35:50.203 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72 4 2024-03-25T19:35:50.228 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:50.228+0000 7f45c3e10040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:35:50.322 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:50.323+0000 7f45c3e10040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:35:50.343 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.343+0000 7f8cbc2c8640 10 monclient: get_monmap_and_config 2024-03-25T19:35:50.343 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.343+0000 7f8cbc2c8640 10 monclient: build_initial_monmap 2024-03-25T19:35:50.343 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.343+0000 7f8cbc2c8640 10 monclient: monmap: 2024-03-25T19:35:50.343 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:50.343 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.343 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:35:50.344601+0000 2024-03-25T19:35:50.343 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:35:50.344601+0000 2024-03-25T19:35:50.343 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:50.343 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.344 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:50.344 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:35:50.344 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:35:50.344 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.344+0000 7f8cbc2c8640 1 Processor -- start 2024-03-25T19:35:50.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.344+0000 7f8cbc2c8640 1 -- start start 2024-03-25T19:35:50.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.344+0000 7f8cbc2c8640 10 monclient: init 2024-03-25T19:35:50.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cbc2c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:50.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cbc2c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:50.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cbc2c8640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb4070210 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:50.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cbc2c8640 10 monclient(hunting): picked mon.noname-a con 0x7f8cb40b06d0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:50.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cbc2c8640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb40705f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:50.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cbc2c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb40b0b80 con 0x7f8cb40b06d0 2024-03-25T19:35:50.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cbc2c8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb40b0430 con 0x7f8cb4070210 2024-03-25T19:35:50.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cbc2c8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb4071670 con 0x7f8cb40705f0 2024-03-25T19:35:50.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cbc2c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:50.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cbc2c8640 10 monclient(hunting): authenticate will time out at 4553.502836s 2024-03-25T19:35:50.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cba064640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f8cb4070210 0x7f8cb4140930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52296/0 (socket says 172.21.15.195:52296) 2024-03-25T19:35:50.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.345+0000 7f8cba064640 1 -- 172.21.15.195:0/2270597462 learned_addr learned my addr 172.21.15.195:0/2270597462 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:50.348 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.347+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163205938 0 0) 0x7f8cb0001110 con 0x7f8cb4070210 2024-03-25T19:35:50.348 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.347+0000 7f8cb9062640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.348 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.347+0000 7f8cb9062640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.348 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.347+0000 7f8cb9062640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.348 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.347+0000 7f8cb9062640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.347+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca0003520 con 0x7f8cb4070210 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369324503 0 0) 0x7f8cb4071670 con 0x7f8cb40705f0 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb0001110 con 0x7f8cb40705f0 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014545396 0 0) 0x7f8cb40b0b80 con 0x7f8cb40b06d0 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:35:50.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.348+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb4071670 con 0x7f8cb40b06d0 2024-03-25T19:35:50.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4196018345 0 0) 0x7f8ca0003520 con 0x7f8cb4070210 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 >> v1:172.21.15.107:6790/0 conn(0x7f8cb40705f0 legacy=0x7f8cb414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 >> v1:172.21.15.107:6789/0 conn(0x7f8cb40b06d0 legacy=0x7f8cb40afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 10 monclient: _finish_hunting 0 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 1 monclient: found mon.noname-c 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb4070a60 con 0x7f8cb4070210 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 10 monclient: _finish_auth 0 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 10 monclient: _check_auth_tickets 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cbc2c8640 5 monclient: authenticate success, global_id 4141 2024-03-25T19:35:50.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8cb0001680 con 0x7f8cb4070210 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cb9062640 10 monclient: dump: 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.352 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.353 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.351+0000 7f8cbc2c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:50.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.352+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cb00017e0 con 0x7f8cb4070210 2024-03-25T19:35:50.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.352+0000 7f8cb9062640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:50.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.352+0000 7f8cb9062640 1 -- 172.21.15.195:0/2270597462 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8cb0001b70 con 0x7f8cb4070210 2024-03-25T19:35:50.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.352+0000 7f8cbc2c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:50.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.352+0000 7f8cbc2c8640 4 set_mon_vals no callback set 2024-03-25T19:35:50.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.352+0000 7f8cb9062640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.357 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.352+0000 7f8cb9062640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:50.357 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.352+0000 7f8cb9062640 10 monclient: dump: 2024-03-25T19:35:50.357 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.357 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.357 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.357 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.357 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.353+0000 7f8cbc2c8640 10 monclient: shutdown 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.353+0000 7f8cbc2c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.353+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/2270597462 >> v1:172.21.15.195:6789/0 conn(0x7f8cb4070210 legacy=0x7f8cb4140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.353+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/2270597462 shutdown_connections 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.353+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/2270597462 >> 172.21.15.195:0/2270597462 conn(0x7f8cb40b5cd0 msgr2=0x7f8cb40b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.353+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/2270597462 shutdown_connections 2024-03-25T19:35:50.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.353+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/2270597462 wait complete. 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.354+0000 7f8cbc2c8640 10 monclient: build_initial_monmap 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.354+0000 7f8cbc2c8640 10 monclient: monmap: 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.354+0000 7f8cbc2c8640 1 librados: starting msgr at 2024-03-25T19:35:50.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.354+0000 7f8cbc2c8640 1 librados: starting objecter 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.354+0000 7f8cbc2c8640 1 Processor -- start 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.354+0000 7f8cbc2c8640 1 -- start start 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.354+0000 7f8cbc2c8640 1 librados: setting wanted keys 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.354+0000 7f8cbc2c8640 1 librados: calling monclient init 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.354+0000 7f8cbc2c8640 10 monclient: init 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cbc2c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cbc2c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cbc2c8640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb4070210 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cbc2c8640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb40705f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cbc2c8640 10 monclient(hunting): picked mon.noname-a con 0x7f8cb40b06d0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cbc2c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb413adf0 con 0x7f8cb40b06d0 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cbc2c8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb41db8e0 con 0x7f8cb40705f0 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cbc2c8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb41db550 con 0x7f8cb4070210 2024-03-25T19:35:50.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cbc2c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:50.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cbc2c8640 10 monclient(hunting): authenticate will time out at 4553.513151s 2024-03-25T19:35:50.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cb9863640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f8cb40705f0 0x7f8cb4131f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52300/0 (socket says 172.21.15.195:52300) 2024-03-25T19:35:50.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.355+0000 7f8cb9863640 1 -- 172.21.15.195:0/1632194013 learned_addr learned my addr 172.21.15.195:0/1632194013 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:50.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.356+0000 7f8caaffd640 10 client.?.objecter ms_handle_connect 0x7f8cb40705f0 2024-03-25T19:35:50.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.356+0000 7f8caaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:50.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.356+0000 7f8caaffd640 10 client.?.objecter ms_handle_connect 0x7f8cb4070210 2024-03-25T19:35:50.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.356+0000 7f8caaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:50.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.356+0000 7f8caaffd640 10 client.?.objecter ms_handle_connect 0x7f8cb40b06d0 2024-03-25T19:35:50.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.356+0000 7f8caaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:50.362 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:50.356+0000 7f18f6e42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:35:50.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.357+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918251914 0 0) 0x7f8cb41db8e0 con 0x7f8cb40705f0 2024-03-25T19:35:50.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:35:50.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c98004eb0 con 0x7f8cb40705f0 2024-03-25T19:35:50.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319737233 0 0) 0x7f8cb41db550 con 0x7f8cb4070210 2024-03-25T19:35:50.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb41db8e0 con 0x7f8cb4070210 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183813090 0 0) 0x7f8cb413adf0 con 0x7f8cb40b06d0 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.358+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb41db550 con 0x7f8cb40b06d0 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.362+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 318793359 0 0) 0x7f8c98004eb0 con 0x7f8cb40705f0 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.362+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb413adf0 con 0x7f8cb40705f0 2024-03-25T19:35:50.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.362+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8ca4001970 con 0x7f8cb40705f0 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.362+0000 7f8caaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.362+0000 7f8caaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.362+0000 7f8caaffd640 10 monclient(hunting): dump: 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.366 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.363+0000 7f8caaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.363+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3373233427 0 0) 0x7f8cb41db8e0 con 0x7f8cb4070210 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.363+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c98004eb0 con 0x7f8cb4070210 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.363+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8cb00012a0 con 0x7f8cb4070210 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.363+0000 7f8caaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.363+0000 7f8caaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.363+0000 7f8caaffd640 10 monclient(hunting): dump: 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.367 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.364+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 987575018 0 0) 0x7f8cb41db550 con 0x7f8cb40b06d0 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.364+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb41db8e0 con 0x7f8cb40b06d0 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.364+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8cac001cb0 con 0x7f8cb40b06d0 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.364+0000 7f8caaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.364+0000 7f8caaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.364+0000 7f8caaffd640 10 monclient(hunting): dump: 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.368 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.369 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.369 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.369 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.369 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.369 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.369 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.369 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3850787094 0 0) 0x7f8cb413adf0 con 0x7f8cb40705f0 2024-03-25T19:35:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 >> v1:172.21.15.107:6790/0 conn(0x7f8cb4070210 legacy=0x7f8cb41dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 >> v1:172.21.15.107:6789/0 conn(0x7f8cb40b06d0 legacy=0x7f8cb41d7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8caaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8caaffd640 1 monclient: found mon.b 2024-03-25T19:35:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8caaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8caaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb4139b70 con 0x7f8cb40705f0 2024-03-25T19:35:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8caaffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8caaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:50.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8caaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8cbc2c8640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8cbc2c8640 10 monclient: _renew_subs 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8cbc2c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/1632194013 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb4139e20 con 0x7f8cb40705f0 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8cbc2c8640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8cbc2c8640 10 monclient: _renew_subs 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8cbc2c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/1632194013 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb4132830 con 0x7f8cb40705f0 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.366+0000 7f8cbc2c8640 1 librados: init done 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.367+0000 7f8cbc2c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.367+0000 7f8cbc2c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.367+0000 7f8cbc2c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:50.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.367+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/1632194013 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb41dbb20 con 0x7f8cb40705f0 2024-03-25T19:35:50.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ca4003610 con 0x7f8cb40705f0 2024-03-25T19:35:50.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:50.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8ca4003880 con 0x7f8cb40705f0 2024-03-25T19:35:50.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:50.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 10 monclient: dump: 2024-03-25T19:35:50.467 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.467 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.467 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8ca4003e30 con 0x7f8cb40705f0 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1849473281 0 0) 0x7f8ca40065a0 con 0x7f8cb40705f0 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 10 client.4144.objecter ms_dispatch 0x7f8cb4000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:35:50.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:35:50.469 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8cb8861640 4 set_mon_vals no callback set 2024-03-25T19:35:50.469 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T19:35:50.469 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8ca4006920 con 0x7f8cb40705f0 2024-03-25T19:35:50.469 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:50.469 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.467+0000 7f8caaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:50.485 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:50.485+0000 7f45c3e10040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:35:50.526 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:50.527+0000 7f45c3e10040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:35:50.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.540+0000 7f8cbc2c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72", "id": 4}] 2024-03-25T19:35:50.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.540+0000 7f8cbc2c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72", "id": 4}] 2024-03-25T19:35:50.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.540+0000 7f8cbc2c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:50.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.540+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/1632194013 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72", "id": 4} v 0) v1 -- 0x7f8cb41db6e0 con 0x7f8cb40705f0 2024-03-25T19:35:50.611 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.611+0000 7f8caaffd640 1 -- 172.21.15.195:0/1632194013 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 4030289944 0 1697012583) 0x7f8ca4006b60 con 0x7f8cb40705f0 2024-03-25T19:35:50.611 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.611+0000 7f8caaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72", "id": 4}] 2024-03-25T19:35:50.611 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.611+0000 7f8caaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:50.611 INFO:teuthology.orchestra.run.smithi195.stdout:4 2024-03-25T19:35:50.612 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 10 librados: watch_flush enter 2024-03-25T19:35:50.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 10 librados: watch_flush exit 2024-03-25T19:35:50.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T19:35:50.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 10 client.4144.objecter successfully canceled tick 2024-03-25T19:35:50.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 10 monclient: shutdown 2024-03-25T19:35:50.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:50.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/1632194013 >> v1:172.21.15.195:6789/0 conn(0x7f8cb40705f0 legacy=0x7f8cb4131f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/1632194013 shutdown_connections 2024-03-25T19:35:50.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/1632194013 >> 172.21.15.195:0/1632194013 conn(0x7f8cb40b5cd0 msgr2=0x7f8cb40afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:50.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/1632194013 shutdown_connections 2024-03-25T19:35:50.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 1 -- 172.21.15.195:0/1632194013 wait complete. 2024-03-25T19:35:50.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.612+0000 7f8cbc2c8640 1 librados: shutdown 2024-03-25T19:35:50.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 5d4d6f31-2da7-4634-9c90-671a6f90572e 5 2024-03-25T19:35:50.698 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:50.698+0000 7f45c3e10040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:35:50.722 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.723+0000 7f6dadc4e640 10 monclient: get_monmap_and_config 2024-03-25T19:35:50.722 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.723+0000 7f6dadc4e640 10 monclient: build_initial_monmap 2024-03-25T19:35:50.722 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.723+0000 7f6dadc4e640 10 monclient: monmap: 2024-03-25T19:35:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:35:50.724084+0000 2024-03-25T19:35:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:35:50.724084+0000 2024-03-25T19:35:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:35:50.723 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:35:50.723 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.723+0000 7f6dadc4e640 1 Processor -- start 2024-03-25T19:35:50.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 1 -- start start 2024-03-25T19:35:50.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 10 monclient: init 2024-03-25T19:35:50.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:50.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:50.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 10 monclient(hunting): picked mon.noname-b con 0x7f6da80b4700 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:50.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 10 monclient(hunting): picked mon.noname-c con 0x7f6da8142a20 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:50.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 10 monclient(hunting): picked mon.noname-a con 0x7f6da80a96b0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:50.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da80a9d60 con 0x7f6da80a96b0 2024-03-25T19:35:50.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da813ff10 con 0x7f6da8142a20 2024-03-25T19:35:50.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da80aa1f0 con 0x7f6da80b4700 2024-03-25T19:35:50.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:50.728 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.724+0000 7f6dadc4e640 10 monclient(hunting): authenticate will time out at 4553.882460s 2024-03-25T19:35:50.728 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.725+0000 7f6da6ffd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6da8142a20 0x7f6da813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52312/0 (socket says 172.21.15.195:52312) 2024-03-25T19:35:50.728 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.725+0000 7f6da6ffd640 1 -- 172.21.15.195:0/3758548487 learned_addr learned my addr 172.21.15.195:0/3758548487 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387181598 0 0) 0x7f6da813ff10 con 0x7f6da8142a20 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d90003520 con 0x7f6da8142a20 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981059344 0 0) 0x7f6da80a9d60 con 0x7f6da80a96b0 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da813ff10 con 0x7f6da80a96b0 2024-03-25T19:35:50.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106264852 0 0) 0x7f6da80aa1f0 con 0x7f6da80b4700 2024-03-25T19:35:50.730 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.730 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.730 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.730 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:35:50.730 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.728+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da80a9d60 con 0x7f6da80b4700 2024-03-25T19:35:50.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1611311433 0 0) 0x7f6d90003520 con 0x7f6da8142a20 2024-03-25T19:35:50.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 >> v1:172.21.15.107:6790/0 conn(0x7f6da80b4700 legacy=0x7f6da8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 >> v1:172.21.15.107:6789/0 conn(0x7f6da80a96b0 legacy=0x7f6da813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:50.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 1 monclient: found mon.noname-c 2024-03-25T19:35:50.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:50.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:35:50.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da80b4ae0 con 0x7f6da8142a20 2024-03-25T19:35:50.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:50.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6dadc4e640 5 monclient: authenticate success, global_id 4147 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6dadc4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6d9c001690 con 0x7f6da8142a20 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6da67fc640 10 monclient: dump: 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.732 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.733 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.733 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.733 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.733 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.731+0000 7f6dadc4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:50.734 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.733+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d9c0017f0 con 0x7f6da8142a20 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.733+0000 7f6da67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.733+0000 7f6da67fc640 1 -- 172.21.15.195:0/3758548487 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6d9c001c00 con 0x7f6da8142a20 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.733+0000 7f6da67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.733+0000 7f6da67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.733+0000 7f6da67fc640 10 monclient: dump: 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.735 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.733+0000 7f6dadc4e640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.733+0000 7f6dadc4e640 4 set_mon_vals no callback set 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.734+0000 7f6dadc4e640 10 monclient: shutdown 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.734+0000 7f6dadc4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.734+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3758548487 >> v1:172.21.15.195:6789/0 conn(0x7f6da8142a20 legacy=0x7f6da813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.734+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3758548487 shutdown_connections 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.734+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3758548487 >> 172.21.15.195:0/3758548487 conn(0x7f6da80668e0 msgr2=0x7f6da8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.734+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3758548487 shutdown_connections 2024-03-25T19:35:50.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.734+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3758548487 wait complete. 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.734+0000 7f6dadc4e640 10 monclient: build_initial_monmap 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.734+0000 7f6dadc4e640 10 monclient: monmap: 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.735+0000 7f6dadc4e640 1 librados: starting msgr at 2024-03-25T19:35:50.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.735+0000 7f6dadc4e640 1 librados: starting objecter 2024-03-25T19:35:50.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.735+0000 7f6dadc4e640 1 Processor -- start 2024-03-25T19:35:50.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.735+0000 7f6dadc4e640 1 -- start start 2024-03-25T19:35:50.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.735+0000 7f6dadc4e640 1 librados: setting wanted keys 2024-03-25T19:35:50.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.735+0000 7f6dadc4e640 1 librados: calling monclient init 2024-03-25T19:35:50.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.735+0000 7f6dadc4e640 10 monclient: init 2024-03-25T19:35:50.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.735+0000 7f6dadc4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:50.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.735+0000 7f6dadc4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:50.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.735+0000 7f6dadc4e640 10 monclient(hunting): picked mon.noname-b con 0x7f6da80a96b0 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:50.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6dadc4e640 10 monclient(hunting): picked mon.noname-a con 0x7f6da80b4700 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:50.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6dadc4e640 10 monclient(hunting): picked mon.noname-c con 0x7f6da8142a20 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:50.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6dadc4e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da81d7590 con 0x7f6da80b4700 2024-03-25T19:35:50.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6dadc4e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da813ccc0 con 0x7f6da80a96b0 2024-03-25T19:35:50.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6dadc4e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da81d8180 con 0x7f6da8142a20 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6dadc4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6dadc4e640 10 monclient(hunting): authenticate will time out at 4553.893800s 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6da77fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6da80a96b0 0x7f6da81d7a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52326/0 (socket says 172.21.15.195:52326) 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6da77fe640 1 -- 172.21.15.195:0/3184435574 learned_addr learned my addr 172.21.15.195:0/3184435574 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6d8ffff640 10 client.?.objecter ms_handle_connect 0x7f6da80a96b0 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6d8ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6d8ffff640 10 client.?.objecter ms_handle_connect 0x7f6da80b4700 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6d8ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6d8ffff640 10 client.?.objecter ms_handle_connect 0x7f6da8142a20 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.736+0000 7f6d8ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247684708 0 0) 0x7f6da813ccc0 con 0x7f6da80a96b0 2024-03-25T19:35:50.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:35:50.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d94004eb0 con 0x7f6da80a96b0 2024-03-25T19:35:50.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130091704 0 0) 0x7f6da81d8180 con 0x7f6da8142a20 2024-03-25T19:35:50.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:35:50.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.738+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da813ccc0 con 0x7f6da8142a20 2024-03-25T19:35:50.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.739+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628754134 0 0) 0x7f6da81d7590 con 0x7f6da80b4700 2024-03-25T19:35:50.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.739+0000 7f6d8ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:50.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.739+0000 7f6d8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:50.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.739+0000 7f6d8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:50.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.739+0000 7f6d8ffff640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:35:50.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.739+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da81d8180 con 0x7f6da80b4700 2024-03-25T19:35:50.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.742+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2187659047 0 0) 0x7f6da813ccc0 con 0x7f6da8142a20 2024-03-25T19:35:50.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.742+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6da81d7590 con 0x7f6da8142a20 2024-03-25T19:35:50.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.742+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6da0001a90 con 0x7f6da8142a20 2024-03-25T19:35:50.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.742+0000 7f6d8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.742+0000 7f6d8ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:50.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.742+0000 7f6d8ffff640 10 monclient(hunting): dump: 2024-03-25T19:35:50.745 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.745 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.745 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.746 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.746 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.746 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.746 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.746 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.747 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.747 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.742+0000 7f6d8ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:50.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.743+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 24166771 0 0) 0x7f6d94004eb0 con 0x7f6da80a96b0 2024-03-25T19:35:50.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.743+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6da813ccc0 con 0x7f6da80a96b0 2024-03-25T19:35:50.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.743+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6d98001a10 con 0x7f6da80a96b0 2024-03-25T19:35:50.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.743+0000 7f6d8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.743+0000 7f6d8ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.743+0000 7f6d8ffff640 10 monclient(hunting): dump: 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.748 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.751 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.746+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 586410011 0 0) 0x7f6da81d8180 con 0x7f6da80b4700 2024-03-25T19:35:50.751 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.746+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d94004eb0 con 0x7f6da80b4700 2024-03-25T19:35:50.751 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.746+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6d9c001b00 con 0x7f6da80b4700 2024-03-25T19:35:50.751 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.746+0000 7f6d8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.751 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.746+0000 7f6d8ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:50.752 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.746+0000 7f6d8ffff640 10 monclient(hunting): dump: 2024-03-25T19:35:50.752 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.752 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.752 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.752 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.752 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.752 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.752 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.752 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.752 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.752 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.746+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 262048350 0 0) 0x7f6da81d7590 con 0x7f6da8142a20 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 >> v1:172.21.15.195:6789/0 conn(0x7f6da80a96b0 legacy=0x7f6da81d7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 >> v1:172.21.15.107:6789/0 conn(0x7f6da80b4700 legacy=0x7f6da81d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6d8ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6d8ffff640 1 monclient: found mon.c 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6d8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6d8ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da81d8310 con 0x7f6da8142a20 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6d8ffff640 10 monclient: _finish_auth 0 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6d8ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6d8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:50.753 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 5 monclient: authenticate success, global_id 4145 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 10 monclient: _renew_subs 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3184435574 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da81d3fd0 con 0x7f6da8142a20 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 10 client.4145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 10 monclient: _renew_subs 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3184435574 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da81d4f40 con 0x7f6da8142a20 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 1 librados: init done 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.747+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3184435574 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da804f730 con 0x7f6da8142a20 2024-03-25T19:35:50.754 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.748+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6da0001fa0 con 0x7f6da8142a20 2024-03-25T19:35:50.755 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.748+0000 7f6d8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:50.755 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.748+0000 7f6da5ffb640 4 set_mon_vals no callback set 2024-03-25T19:35:50.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6da0003b10 con 0x7f6da8142a20 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 10 monclient: dump: 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:50.844 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:50.845 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:50.845 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6da0004000 con 0x7f6da8142a20 2024-03-25T19:35:50.845 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 346431822 0 0) 0x7f6da00056e0 con 0x7f6da8142a20 2024-03-25T19:35:50.845 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 10 client.4145.objecter ms_dispatch 0x7f6da8000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:35:50.845 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 3 client.4145.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:35:50.845 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 3 client.4145.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:35:50.845 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 20 client.4145.objecter dump_active .. 0 homeless 2024-03-25T19:35:50.846 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6da0005270 con 0x7f6da8142a20 2024-03-25T19:35:50.846 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:50.846 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.844+0000 7f6d8ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:50.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.919+0000 7f6dadc4e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5d4d6f31-2da7-4634-9c90-671a6f90572e", "id": 5}] 2024-03-25T19:35:50.919 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.919+0000 7f6dadc4e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5d4d6f31-2da7-4634-9c90-671a6f90572e", "id": 5}] 2024-03-25T19:35:50.919 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.919+0000 7f6dadc4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:50.919 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.919+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3184435574 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd new", "uuid": "5d4d6f31-2da7-4634-9c90-671a6f90572e", "id": 5} v 0) v1 -- 0x7f6da81d7760 con 0x7f6da8142a20 2024-03-25T19:35:50.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.954+0000 7f6d8ffff640 1 -- 172.21.15.195:0/3184435574 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5d4d6f31-2da7-4634-9c90-671a6f90572e", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 209637005 0 1988412176) 0x7f6da0005500 con 0x7f6da8142a20 2024-03-25T19:35:50.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.954+0000 7f6d8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5d4d6f31-2da7-4634-9c90-671a6f90572e", "id": 5}] 2024-03-25T19:35:50.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.954+0000 7f6d8ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:50.954 INFO:teuthology.orchestra.run.smithi195.stdout:5 2024-03-25T19:35:50.955 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 10 librados: watch_flush enter 2024-03-25T19:35:50.955 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 10 librados: watch_flush exit 2024-03-25T19:35:50.955 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 20 client.4145.objecter shutdown clearing up homeless session... 2024-03-25T19:35:50.955 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 10 client.4145.objecter successfully canceled tick 2024-03-25T19:35:50.955 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 10 monclient: shutdown 2024-03-25T19:35:50.955 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:50.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3184435574 >> v1:172.21.15.107:6790/0 conn(0x7f6da8142a20 legacy=0x7f6da813b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3184435574 shutdown_connections 2024-03-25T19:35:50.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3184435574 >> 172.21.15.195:0/3184435574 conn(0x7f6da80668e0 msgr2=0x7f6da80b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:50.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3184435574 shutdown_connections 2024-03-25T19:35:50.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 1 -- 172.21.15.195:0/3184435574 wait complete. 2024-03-25T19:35:50.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:50.955+0000 7f6dadc4e640 1 librados: shutdown 2024-03-25T19:35:50.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new abea285a-e8c1-45fc-a93e-f9e979553b41 6 2024-03-25T19:35:51.066 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.066+0000 7f6f07c86640 10 monclient: get_monmap_and_config 2024-03-25T19:35:51.066 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.066+0000 7f6f07c86640 10 monclient: build_initial_monmap 2024-03-25T19:35:51.066 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.066+0000 7f6f07c86640 10 monclient: monmap: 2024-03-25T19:35:51.066 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:51.066 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.066 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:35:51.067256+0000 2024-03-25T19:35:51.066 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:35:51.067256+0000 2024-03-25T19:35:51.067 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:51.067 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.067 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:51.067 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:35:51.067 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:35:51.067 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.067 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.066+0000 7f6f07c86640 1 Processor -- start 2024-03-25T19:35:51.067 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 1 -- start start 2024-03-25T19:35:51.067 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 10 monclient: init 2024-03-25T19:35:51.068 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:51.068 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:51.068 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 10 monclient(hunting): picked mon.noname-a con 0x7f6f000b4700 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:51.069 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 10 monclient(hunting): picked mon.noname-b con 0x7f6f00142a20 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:51.069 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 10 monclient(hunting): picked mon.noname-c con 0x7f6f000a96b0 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:51.069 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000a9d60 con 0x7f6f000b4700 2024-03-25T19:35:51.069 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0013ff10 con 0x7f6f000a96b0 2024-03-25T19:35:51.069 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000aa1f0 con 0x7f6f00142a20 2024-03-25T19:35:51.069 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:51.069 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.067+0000 7f6f07c86640 10 monclient(hunting): authenticate will time out at 4554.225526s 2024-03-25T19:35:51.069 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.068+0000 7f6f06223640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6f000a96b0 0x7f6f0013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52336/0 (socket says 172.21.15.195:52336) 2024-03-25T19:35:51.069 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.068+0000 7f6f06223640 1 -- 172.21.15.195:0/1494268024 learned_addr learned my addr 172.21.15.195:0/1494268024 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:51.070 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495368923 0 0) 0x7f6f0013ff10 con 0x7f6f000a96b0 2024-03-25T19:35:51.070 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.070 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eec003520 con 0x7f6f000a96b0 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056464100 0 0) 0x7f6f000a9d60 con 0x7f6f000b4700 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f0013ff10 con 0x7f6f000b4700 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621176562 0 0) 0x7f6f000aa1f0 con 0x7f6f00142a20 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:35:51.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.070+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f000a9d60 con 0x7f6f00142a20 2024-03-25T19:35:51.073 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4040729893 0 0) 0x7f6eec003520 con 0x7f6f000a96b0 2024-03-25T19:35:51.073 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 >> v1:172.21.15.107:6790/0 conn(0x7f6f00142a20 legacy=0x7f6f0013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.073 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 >> v1:172.21.15.107:6789/0 conn(0x7f6f000b4700 legacy=0x7f6f00142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 10 monclient: _finish_hunting 0 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 1 monclient: found mon.noname-c 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f000b4ae0 con 0x7f6f000a96b0 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 10 monclient: _finish_auth 0 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 10 monclient: _check_auth_tickets 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6ef8001a10 con 0x7f6f000a96b0 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.073+0000 7f6f04a20640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:51.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.074+0000 7f6f04a20640 10 monclient: dump: 2024-03-25T19:35:51.075 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.075 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.075 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.075 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.075 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.075 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.075 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.075 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.075 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.075 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.074+0000 7f6f07c86640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:35:51.076 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.074+0000 7f6f07c86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.075+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ef8001b70 con 0x7f6f000a96b0 2024-03-25T19:35:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.075+0000 7f6f04a20640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.075+0000 7f6f04a20640 1 -- 172.21.15.195:0/1494268024 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6ef8001f80 con 0x7f6f000a96b0 2024-03-25T19:35:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.075+0000 7f6f07c86640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.075+0000 7f6f07c86640 4 set_mon_vals no callback set 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.075+0000 7f6f04a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.075+0000 7f6f04a20640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.075+0000 7f6f04a20640 10 monclient: dump: 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.080 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.076+0000 7f6f07c86640 10 monclient: shutdown 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.076+0000 7f6f07c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.076+0000 7f6f07c86640 1 -- 172.21.15.195:0/1494268024 >> v1:172.21.15.195:6789/0 conn(0x7f6f000a96b0 legacy=0x7f6f0013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.076+0000 7f6f07c86640 1 -- 172.21.15.195:0/1494268024 shutdown_connections 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.076+0000 7f6f07c86640 1 -- 172.21.15.195:0/1494268024 >> 172.21.15.195:0/1494268024 conn(0x7f6f000668e0 msgr2=0x7f6f00066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.076+0000 7f6f07c86640 1 -- 172.21.15.195:0/1494268024 shutdown_connections 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.076+0000 7f6f07c86640 1 -- 172.21.15.195:0/1494268024 wait complete. 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.077+0000 7f6f07c86640 10 monclient: build_initial_monmap 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.077+0000 7f6f07c86640 10 monclient: monmap: 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-25T19:35:51.081 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-25T19:35:51.082 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:51.082 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-25T19:35:51.082 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:51.082 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:35:51.082 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:35:51.082 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.082 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.077+0000 7f6f07c86640 1 librados: starting msgr at 2024-03-25T19:35:51.082 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.077+0000 7f6f07c86640 1 librados: starting objecter 2024-03-25T19:35:51.082 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.077+0000 7f6f07c86640 1 Processor -- start 2024-03-25T19:35:51.083 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.077+0000 7f6f07c86640 1 -- start start 2024-03-25T19:35:51.083 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.077+0000 7f6f07c86640 1 librados: setting wanted keys 2024-03-25T19:35:51.083 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.077+0000 7f6f07c86640 1 librados: calling monclient init 2024-03-25T19:35:51.083 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.077+0000 7f6f07c86640 10 monclient: init 2024-03-25T19:35:51.083 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f07c86640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:51.083 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f07c86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:51.083 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f07c86640 10 monclient(hunting): picked mon.noname-b con 0x7f6f000a96b0 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:51.083 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f07c86640 10 monclient(hunting): picked mon.noname-a con 0x7f6f000b4700 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:51.083 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f07c86640 10 monclient(hunting): picked mon.noname-c con 0x7f6f00142a20 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:51.083 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f07c86640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000add10 con 0x7f6f000b4700 2024-03-25T19:35:51.083 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f07c86640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000acb70 con 0x7f6f000a96b0 2024-03-25T19:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f07c86640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0014e7e0 con 0x7f6f00142a20 2024-03-25T19:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f07c86640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f07c86640 10 monclient(hunting): authenticate will time out at 4554.235895s 2024-03-25T19:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f05a22640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6f000a96b0 0x7f6f000ac370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52352/0 (socket says 172.21.15.195:52352) 2024-03-25T19:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6f05a22640 1 -- 172.21.15.195:0/352622027 learned_addr learned my addr 172.21.15.195:0/352622027 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6ef67fc640 10 client.?.objecter ms_handle_connect 0x7f6f000a96b0 2024-03-25T19:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.078+0000 7f6ef67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.079+0000 7f6ef67fc640 10 client.?.objecter ms_handle_connect 0x7f6f000b4700 2024-03-25T19:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.079+0000 7f6ef67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.079+0000 7f6ef67fc640 10 client.?.objecter ms_handle_connect 0x7f6f00142a20 2024-03-25T19:35:51.084 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.079+0000 7f6ef67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:51.086 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816677113 0 0) 0x7f6f000acb70 con 0x7f6f000a96b0 2024-03-25T19:35:51.086 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.086 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.086 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.086 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:35:51.086 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee4004eb0 con 0x7f6f000a96b0 2024-03-25T19:35:51.086 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69346373 0 0) 0x7f6f0014e7e0 con 0x7f6f00142a20 2024-03-25T19:35:51.086 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f000acb70 con 0x7f6f00142a20 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378417617 0 0) 0x7f6f000add10 con 0x7f6f000b4700 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.081+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f0014e7e0 con 0x7f6f000b4700 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.085+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 703727081 0 0) 0x7f6f000acb70 con 0x7f6f00142a20 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f000add10 con 0x7f6f00142a20 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6ef8001e80 con 0x7f6f00142a20 2024-03-25T19:35:51.087 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 10 monclient(hunting): dump: 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.088 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1124588824 0 0) 0x7f6ee4004eb0 con 0x7f6f000a96b0 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f000acb70 con 0x7f6f000a96b0 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6efc001a30 con 0x7f6f000a96b0 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.086+0000 7f6ef67fc640 10 monclient(hunting): dump: 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.089 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.090 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.090 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.090 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.090 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.091 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.087+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2222549334 0 0) 0x7f6f0014e7e0 con 0x7f6f000b4700 2024-03-25T19:35:51.091 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ee4004eb0 con 0x7f6f000b4700 2024-03-25T19:35:51.091 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6ef0001790 con 0x7f6f000b4700 2024-03-25T19:35:51.091 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.091 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:51.091 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 10 monclient(hunting): dump: 2024-03-25T19:35:51.091 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 449831236 0 0) 0x7f6f000acb70 con 0x7f6f000a96b0 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 >> v1:172.21.15.107:6790/0 conn(0x7f6f00142a20 legacy=0x7f6f001daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.092 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 >> v1:172.21.15.107:6789/0 conn(0x7f6f000b4700 legacy=0x7f6f00136060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.093 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:51.093 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 1 monclient: found mon.b 2024-03-25T19:35:51.093 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:51.093 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.093 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f000ab760 con 0x7f6f000a96b0 2024-03-25T19:35:51.093 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:51.093 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:51.093 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6ef67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:51.093 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6f07c86640 5 monclient: authenticate success, global_id 4156 2024-03-25T19:35:51.093 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.088+0000 7f6f07c86640 10 monclient: _renew_subs 2024-03-25T19:35:51.093 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.089+0000 7f6f07c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.094 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.089+0000 7f6f07c86640 1 -- 172.21.15.195:0/352622027 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f00136770 con 0x7f6f000a96b0 2024-03-25T19:35:51.094 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.089+0000 7f6f07c86640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:51.094 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.089+0000 7f6f07c86640 10 monclient: _renew_subs 2024-03-25T19:35:51.094 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.089+0000 7f6f07c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.094 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.089+0000 7f6f07c86640 1 -- 172.21.15.195:0/352622027 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f001db380 con 0x7f6f000a96b0 2024-03-25T19:35:51.094 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.089+0000 7f6f07c86640 1 librados: init done 2024-03-25T19:35:51.094 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.089+0000 7f6f07c86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:51.094 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.089+0000 7f6f07c86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:51.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.089+0000 7f6f07c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.089+0000 7f6f07c86640 1 -- 172.21.15.195:0/352622027 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f000acd00 con 0x7f6f000a96b0 2024-03-25T19:35:51.189 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f18f6e42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:35:51.190 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6efc001c50 con 0x7f6f000a96b0 2024-03-25T19:35:51.190 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6efc003a60 con 0x7f6f000a96b0 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 10 monclient: dump: 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.191 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.192 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6efc004010 con 0x7f6f000a96b0 2024-03-25T19:35:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef7fff640 4 set_mon_vals no callback set 2024-03-25T19:35:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3391629735 0 0) 0x7f6efc006810 con 0x7f6f000a96b0 2024-03-25T19:35:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 10 client.4156.objecter ms_dispatch 0x7f6f00000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:35:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:35:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:35:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T19:35:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6efc018de0 con 0x7f6f000a96b0 2024-03-25T19:35:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:51.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.190+0000 7f6ef67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:51.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.263+0000 7f6f07c86640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "abea285a-e8c1-45fc-a93e-f9e979553b41", "id": 6}] 2024-03-25T19:35:51.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.264+0000 7f6f07c86640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "abea285a-e8c1-45fc-a93e-f9e979553b41", "id": 6}] 2024-03-25T19:35:51.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.264+0000 7f6f07c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.264+0000 7f6f07c86640 1 -- 172.21.15.195:0/352622027 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd new", "uuid": "abea285a-e8c1-45fc-a93e-f9e979553b41", "id": 6} v 0) v1 -- 0x7f6f000ae1f0 con 0x7f6f000a96b0 2024-03-25T19:35:51.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.325+0000 7f6ef67fc640 1 -- 172.21.15.195:0/352622027 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "abea285a-e8c1-45fc-a93e-f9e979553b41", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1997545643 0 1113811849) 0x7f6efc0062f0 con 0x7f6f000a96b0 2024-03-25T19:35:51.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.325+0000 7f6ef67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "abea285a-e8c1-45fc-a93e-f9e979553b41", "id": 6}] 2024-03-25T19:35:51.325 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.325+0000 7f6ef67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:51.325 INFO:teuthology.orchestra.run.smithi195.stdout:6 2024-03-25T19:35:51.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 10 librados: watch_flush enter 2024-03-25T19:35:51.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 10 librados: watch_flush exit 2024-03-25T19:35:51.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T19:35:51.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 10 client.4156.objecter successfully canceled tick 2024-03-25T19:35:51.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 10 monclient: shutdown 2024-03-25T19:35:51.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:51.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 1 -- 172.21.15.195:0/352622027 >> v1:172.21.15.195:6789/0 conn(0x7f6f000a96b0 legacy=0x7f6f000ac370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 1 -- 172.21.15.195:0/352622027 shutdown_connections 2024-03-25T19:35:51.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 1 -- 172.21.15.195:0/352622027 >> 172.21.15.195:0/352622027 conn(0x7f6f000668e0 msgr2=0x7f6f0013c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:51.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 1 -- 172.21.15.195:0/352622027 shutdown_connections 2024-03-25T19:35:51.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 1 -- 172.21.15.195:0/352622027 wait complete. 2024-03-25T19:35:51.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.326+0000 7f6f07c86640 1 librados: shutdown 2024-03-25T19:35:51.336 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new bfc712bb-31d4-489b-bd81-49e6acbac4e1 7 2024-03-25T19:35:51.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.436+0000 7f4b898e0640 10 monclient: get_monmap_and_config 2024-03-25T19:35:51.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.436+0000 7f4b898e0640 10 monclient: build_initial_monmap 2024-03-25T19:35:51.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.436+0000 7f4b898e0640 10 monclient: monmap: 2024-03-25T19:35:51.436 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:51.436 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.436 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:35:51.437438+0000 2024-03-25T19:35:51.436 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:35:51.437438+0000 2024-03-25T19:35:51.436 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:51.436 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.436 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:51.437 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:35:51.437 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:35:51.437 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 1 Processor -- start 2024-03-25T19:35:51.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 1 -- start start 2024-03-25T19:35:51.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 10 monclient: init 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 10 monclient(hunting): picked mon.noname-a con 0x7f4b8406c2d0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 10 monclient(hunting): picked mon.noname-b con 0x7f4b8406c6b0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 10 monclient(hunting): picked mon.noname-c con 0x7f4b84142d90 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b840b0840 con 0x7f4b8406c2d0 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b84143200 con 0x7f4b84142d90 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8406d810 con 0x7f4b8406c6b0 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.437+0000 7f4b898e0640 10 monclient(hunting): authenticate will time out at 4554.595520s 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.438+0000 7f4b837fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4b84142d90 0x7f4b8414dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52358/0 (socket says 172.21.15.195:52358) 2024-03-25T19:35:51.439 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.438+0000 7f4b837fe640 1 -- 172.21.15.195:0/4097577299 learned_addr learned my addr 172.21.15.195:0/4097577299 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:51.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861798982 0 0) 0x7f4b7c001490 con 0x7f4b84142d90 2024-03-25T19:35:51.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:35:51.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b6c003520 con 0x7f4b84142d90 2024-03-25T19:35:51.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245196100 0 0) 0x7f4b840b0840 con 0x7f4b8406c2d0 2024-03-25T19:35:51.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:35:51.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.440+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b7c001490 con 0x7f4b8406c2d0 2024-03-25T19:35:51.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.441+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607780772 0 0) 0x7f4b8406d810 con 0x7f4b8406c6b0 2024-03-25T19:35:51.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.441+0000 7f4b81ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.441+0000 7f4b81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.441+0000 7f4b81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.441+0000 7f4b81ffb640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:35:51.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.441+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b840b0840 con 0x7f4b8406c6b0 2024-03-25T19:35:51.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2633409969 0 0) 0x7f4b6c003520 con 0x7f4b84142d90 2024-03-25T19:35:51.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 >> v1:172.21.15.107:6790/0 conn(0x7f4b8406c6b0 legacy=0x7f4b84142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 >> v1:172.21.15.107:6789/0 conn(0x7f4b8406c2d0 legacy=0x7f4b840afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:51.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 1 monclient: found mon.noname-c 2024-03-25T19:35:51.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:51.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b840ae260 con 0x7f4b84142d90 2024-03-25T19:35:51.443 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b898e0640 5 monclient: authenticate success, global_id 4159 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4b7c001a00 con 0x7f4b84142d90 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b898e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b81ffb640 10 monclient: dump: 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.444 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.445 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.445 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.445 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.445 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.445 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.445 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.445 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.443+0000 7f4b898e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:51.448 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b7c001be0 con 0x7f4b84142d90 2024-03-25T19:35:51.449 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:51.449 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b81ffb640 1 -- 172.21.15.195:0/4097577299 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4b7c001f70 con 0x7f4b84142d90 2024-03-25T19:35:51.449 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b898e0640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:51.449 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b898e0640 4 set_mon_vals no callback set 2024-03-25T19:35:51.449 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.449 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:51.449 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b81ffb640 10 monclient: dump: 2024-03-25T19:35:51.449 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.449 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.449 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.449 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.450 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.450 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.450 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.450 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.450 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.450 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b898e0640 10 monclient: shutdown 2024-03-25T19:35:51.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b898e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:51.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b898e0640 1 -- 172.21.15.195:0/4097577299 >> v1:172.21.15.195:6789/0 conn(0x7f4b84142d90 legacy=0x7f4b8414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b898e0640 1 -- 172.21.15.195:0/4097577299 shutdown_connections 2024-03-25T19:35:51.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.445+0000 7f4b898e0640 1 -- 172.21.15.195:0/4097577299 >> 172.21.15.195:0/4097577299 conn(0x7f4b840b5e40 msgr2=0x7f4b840b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.446+0000 7f4b898e0640 1 -- 172.21.15.195:0/4097577299 shutdown_connections 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.446+0000 7f4b898e0640 1 -- 172.21.15.195:0/4097577299 wait complete. 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.446+0000 7f4b898e0640 10 monclient: build_initial_monmap 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.446+0000 7f4b898e0640 10 monclient: monmap: 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:35:51.451 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.446+0000 7f4b898e0640 1 librados: starting msgr at 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.446+0000 7f4b898e0640 1 librados: starting objecter 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 1 Processor -- start 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 1 -- start start 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 1 librados: setting wanted keys 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 1 librados: calling monclient init 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 10 monclient: init 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 10 monclient(hunting): picked mon.noname-b con 0x7f4b8406c2d0 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 10 monclient(hunting): picked mon.noname-a con 0x7f4b8406c6b0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 10 monclient(hunting): picked mon.noname-c con 0x7f4b84142d90 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:51.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b84143d40 con 0x7f4b8406c6b0 2024-03-25T19:35:51.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b84070470 con 0x7f4b8406c2d0 2024-03-25T19:35:51.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b84071210 con 0x7f4b84142d90 2024-03-25T19:35:51.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:51.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.447+0000 7f4b898e0640 10 monclient(hunting): authenticate will time out at 4554.605630s 2024-03-25T19:35:51.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.448+0000 7f4b82ffd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4b8406c2d0 0x7f4b84070b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:52368/0 (socket says 172.21.15.195:52368) 2024-03-25T19:35:51.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.448+0000 7f4b82ffd640 1 -- 172.21.15.195:0/2593146025 learned_addr learned my addr 172.21.15.195:0/2593146025 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-25T19:35:51.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.448+0000 7f4b6bfff640 10 client.?.objecter ms_handle_connect 0x7f4b8406c2d0 2024-03-25T19:35:51.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.448+0000 7f4b6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:51.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.448+0000 7f4b6bfff640 10 client.?.objecter ms_handle_connect 0x7f4b8406c6b0 2024-03-25T19:35:51.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.448+0000 7f4b6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:51.454 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.448+0000 7f4b6bfff640 10 client.?.objecter ms_handle_connect 0x7f4b84142d90 2024-03-25T19:35:51.454 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.448+0000 7f4b6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:51.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780077640 0 0) 0x7f4b84070470 con 0x7f4b8406c2d0 2024-03-25T19:35:51.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 10 monclient(hunting): my global_id is 4162 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b70004eb0 con 0x7f4b8406c2d0 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655980481 0 0) 0x7f4b84143d40 con 0x7f4b8406c6b0 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.450+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b84070470 con 0x7f4b8406c6b0 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.451+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270195913 0 0) 0x7f4b84071210 con 0x7f4b84142d90 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.451+0000 7f4b6bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.451+0000 7f4b6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.451+0000 7f4b6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:51.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.451+0000 7f4b6bfff640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.451+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b84143d40 con 0x7f4b84142d90 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.454+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 553470751 0 0) 0x7f4b70004eb0 con 0x7f4b8406c2d0 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b84071210 con 0x7f4b8406c2d0 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4b740019b0 con 0x7f4b8406c2d0 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 10 monclient(hunting): dump: 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.458 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 421371319 0 0) 0x7f4b84070470 con 0x7f4b8406c6b0 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b70004eb0 con 0x7f4b8406c6b0 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4b780018b0 con 0x7f4b8406c6b0 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.455+0000 7f4b6bfff640 10 monclient(hunting): dump: 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.459 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.460 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.460 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.460 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.460 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.460 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.460 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.460 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.460 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2589367699 0 0) 0x7f4b84143d40 con 0x7f4b84142d90 2024-03-25T19:35:51.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b84070470 con 0x7f4b84142d90 2024-03-25T19:35:51.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4b7c001620 con 0x7f4b84142d90 2024-03-25T19:35:51.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:51.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 10 monclient(hunting): dump: 2024-03-25T19:35:51.463 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.463 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.463 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2901075545 0 0) 0x7f4b84071210 con 0x7f4b8406c2d0 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 >> v1:172.21.15.107:6790/0 conn(0x7f4b84142d90 legacy=0x7f4b841e1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 >> v1:172.21.15.107:6789/0 conn(0x7f4b8406c6b0 legacy=0x7f4b841400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 1 monclient: found mon.b 2024-03-25T19:35:51.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b841417c0 con 0x7f4b8406c2d0 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 10 monclient: _finish_auth 0 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b898e0640 5 monclient: authenticate success, global_id 4162 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b898e0640 10 monclient: _renew_subs 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b898e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b898e0640 1 -- 172.21.15.195:0/2593146025 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8413fbb0 con 0x7f4b8406c2d0 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b898e0640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b898e0640 10 monclient: _renew_subs 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b898e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b898e0640 1 -- 172.21.15.195:0/2593146025 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b84134220 con 0x7f4b8406c2d0 2024-03-25T19:35:51.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.458+0000 7f4b898e0640 1 librados: init done 2024-03-25T19:35:51.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.459+0000 7f4b898e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:51.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.459+0000 7f4b898e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:51.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.459+0000 7f4b898e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.459+0000 7f4b898e0640 1 -- 172.21.15.195:0/2593146025 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b84070600 con 0x7f4b8406c2d0 2024-03-25T19:35:51.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b74003610 con 0x7f4b8406c2d0 2024-03-25T19:35:51.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:51.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4b740039a0 con 0x7f4b8406c2d0 2024-03-25T19:35:51.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:51.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 10 monclient: dump: 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4b74003f50 con 0x7f4b8406c2d0 2024-03-25T19:35:51.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b817fa640 4 set_mon_vals no callback set 2024-03-25T19:35:51.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1251231578 0 0) 0x7f4b740068a0 con 0x7f4b8406c2d0 2024-03-25T19:35:51.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 10 client.4162.objecter ms_dispatch 0x7f4b84000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:35:51.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:35:51.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:35:51.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.462+0000 7f4b6bfff640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T19:35:51.484 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:51.485+0000 7f18f6e42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:35:51.537 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:51.537+0000 7f18f6e42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:35:51.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.556+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4b74004200 con 0x7f4b8406c2d0 2024-03-25T19:35:51.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.556+0000 7f4b6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:51.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.556+0000 7f4b6bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:51.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.628+0000 7f4b898e0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bfc712bb-31d4-489b-bd81-49e6acbac4e1", "id": 7}] 2024-03-25T19:35:51.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.628+0000 7f4b898e0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bfc712bb-31d4-489b-bd81-49e6acbac4e1", "id": 7}] 2024-03-25T19:35:51.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.628+0000 7f4b898e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:35:51.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.628+0000 7f4b898e0640 1 -- 172.21.15.195:0/2593146025 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bfc712bb-31d4-489b-bd81-49e6acbac4e1", "id": 7} v 0) v1 -- 0x7f4b8404f730 con 0x7f4b8406c2d0 2024-03-25T19:35:51.656 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:51.657+0000 7f18f6e42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:35:51.675 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:51.675+0000 7f45c3e10040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:35:51.688 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.688+0000 7f4b6bfff640 1 -- 172.21.15.195:0/2593146025 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bfc712bb-31d4-489b-bd81-49e6acbac4e1", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2478272639 0 1371665406) 0x7f4b74006ad0 con 0x7f4b8406c2d0 2024-03-25T19:35:51.688 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.688+0000 7f4b6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bfc712bb-31d4-489b-bd81-49e6acbac4e1", "id": 7}] 2024-03-25T19:35:51.688 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.688+0000 7f4b6bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:51.688 INFO:teuthology.orchestra.run.smithi195.stdout:7 2024-03-25T19:35:51.689 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 10 librados: watch_flush enter 2024-03-25T19:35:51.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 10 librados: watch_flush exit 2024-03-25T19:35:51.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T19:35:51.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 10 client.4162.objecter successfully canceled tick 2024-03-25T19:35:51.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 10 monclient: shutdown 2024-03-25T19:35:51.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:51.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 1 -- 172.21.15.195:0/2593146025 >> v1:172.21.15.195:6789/0 conn(0x7f4b8406c2d0 legacy=0x7f4b84070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:51.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 1 -- 172.21.15.195:0/2593146025 shutdown_connections 2024-03-25T19:35:51.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 1 -- 172.21.15.195:0/2593146025 >> 172.21.15.195:0/2593146025 conn(0x7f4b840b5e40 msgr2=0x7f4b84074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:51.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 1 -- 172.21.15.195:0/2593146025 shutdown_connections 2024-03-25T19:35:51.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 1 -- 172.21.15.195:0/2593146025 wait complete. 2024-03-25T19:35:51.691 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-25T19:35:51.689+0000 7f4b898e0640 1 librados: shutdown 2024-03-25T19:35:51.699 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:51.699 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:35:51.699 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:35:51.702 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:35:51.702 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:51.702 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:35:51.702 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:35:51.704 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:35:51.704 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:51.705 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:35:51.705 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:35:51.707 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:35:51.707 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:51.707 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:35:51.707 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:35:51.709 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:35:51.709 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:51.710 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:35:51.710 DEBUG:teuthology.orchestra.run.smithi195:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:35:51.712 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:35:51.712 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:51.712 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:35:51.712 DEBUG:teuthology.orchestra.run.smithi195:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:35:51.714 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:35:51.715 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:51.715 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:35:51.715 DEBUG:teuthology.orchestra.run.smithi195:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:35:51.717 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:35:51.717 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:51.717 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:35:51.717 DEBUG:teuthology.orchestra.run.smithi195:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:35:51.719 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:35:51.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:51.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.884+0000 7f77442e4640 10 monclient: get_monmap_and_config 2024-03-25T19:35:51.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.884+0000 7f77442e4640 10 monclient: build_initial_monmap 2024-03-25T19:35:51.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.884+0000 7f77442e4640 10 monclient: monmap: 2024-03-25T19:35:51.886 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2024-03-25T19:35:51.886 INFO:teuthology.orchestra.run.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:51.886 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-25T19:35:51.885759+0000 2024-03-25T19:35:51.886 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-25T19:35:51.885759+0000 2024-03-25T19:35:51.886 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:51.886 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:51.886 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:51.886 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:35:51.887 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:35:51.887 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:51.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.886+0000 7f77442e4640 1 Processor -- start 2024-03-25T19:35:51.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.886+0000 7f77442e4640 1 -- start start 2024-03-25T19:35:51.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.886+0000 7f77442e4640 10 monclient: init 2024-03-25T19:35:51.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.887+0000 7f77442e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:51.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.887+0000 7f77442e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:51.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.887+0000 7f77442e4640 10 monclient(hunting): picked mon.noname-c con 0x7f773c0a9cb0 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:51.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.887+0000 7f77442e4640 10 monclient(hunting): picked mon.noname-a con 0x7f773c0aed90 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:51.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.887+0000 7f77442e4640 10 monclient(hunting): picked mon.noname-b con 0x7f773c0acd10 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:51.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.887+0000 7f77442e4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c0ab9b0 con 0x7f773c0aed90 2024-03-25T19:35:51.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.887+0000 7f77442e4640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c0abc20 con 0x7f773c0a9cb0 2024-03-25T19:35:51.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.887+0000 7f77442e4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c0abf00 con 0x7f773c0acd10 2024-03-25T19:35:51.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.887+0000 7f77442e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:51.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.887+0000 7f77442e4640 10 monclient(hunting): authenticate will time out at 4544.227996s 2024-03-25T19:35:51.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.916+0000 7f774187f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f773c0aed90 0x7f773c0ac600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46856/0 (socket says 172.21.15.107:46856) 2024-03-25T19:35:51.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:51.916+0000 7f774187f640 1 -- 172.21.15.107:0/3983979133 learned_addr learned my addr 172.21.15.107:0/3983979133 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:35:52.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.012+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008871484 0 0) 0x7f773c0abc20 con 0x7f773c0a9cb0 2024-03-25T19:35:52.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.012+0000 7f774107e640 10 monclient(hunting): _negotiate 2024-03-25T19:35:52.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.012+0000 7f774107e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:52.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.012+0000 7f774107e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:52.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.012+0000 7f774107e640 10 monclient(hunting): my global_id is 4165 2024-03-25T19:35:52.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.012+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7728003520 con 0x7f773c0a9cb0 2024-03-25T19:35:52.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.017+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543908949 0 0) 0x7f773c0ab9b0 con 0x7f773c0aed90 2024-03-25T19:35:52.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.017+0000 7f774107e640 10 monclient(hunting): _negotiate 2024-03-25T19:35:52.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.017+0000 7f774107e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:52.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.017+0000 7f774107e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:52.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.017+0000 7f774107e640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:35:52.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.017+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f773c0abc20 con 0x7f773c0aed90 2024-03-25T19:35:52.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.018+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741164891 0 0) 0x7f773c0abf00 con 0x7f773c0acd10 2024-03-25T19:35:52.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.018+0000 7f774107e640 10 monclient(hunting): _negotiate 2024-03-25T19:35:52.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.018+0000 7f774107e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:52.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.018+0000 7f774107e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:52.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.018+0000 7f774107e640 10 monclient(hunting): my global_id is 4160 2024-03-25T19:35:52.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.018+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f773c0ab9b0 con 0x7f773c0acd10 2024-03-25T19:35:52.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3029524471 0 0) 0x7f773c0ab9b0 con 0x7f773c0acd10 2024-03-25T19:35:52.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 >> v1:172.21.15.195:6789/0 conn(0x7f773c0a9cb0 legacy=0x7f773c0ae680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 >> v1:172.21.15.107:6789/0 conn(0x7f773c0aed90 legacy=0x7f773c0ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f774107e640 10 monclient: _finish_hunting 0 2024-03-25T19:35:52.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f774107e640 1 monclient: found mon.noname-b 2024-03-25T19:35:52.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f774107e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:52.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f774107e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:35:52.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c0ac1e0 con 0x7f773c0acd10 2024-03-25T19:35:52.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f774107e640 10 monclient: _finish_auth 0 2024-03-25T19:35:52.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f774107e640 10 monclient: _check_auth_tickets 2024-03-25T19:35:52.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f774107e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:52.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f77442e4640 5 monclient: authenticate success, global_id 4160 2024-03-25T19:35:52.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.026+0000 7f77442e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:52.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.040+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f772c0015f0 con 0x7f773c0acd10 2024-03-25T19:35:52.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.040+0000 7f774107e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:52.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.040+0000 7f774107e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:52.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.040+0000 7f774107e640 10 monclient: dump: 2024-03-25T19:35:52.041 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-25T19:35:52.041 INFO:teuthology.orchestra.run.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:52.041 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:52.041 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:52.041 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:52.041 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:52.042 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:52.042 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:52.042 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:52.042 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:52.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.040+0000 7f77442e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:52.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.045+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f772c001880 con 0x7f773c0acd10 2024-03-25T19:35:52.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.045+0000 7f774107e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:52.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.045+0000 7f77442e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:52.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.045+0000 7f77442e4640 4 set_mon_vals no callback set 2024-03-25T19:35:52.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.046+0000 7f77442e4640 10 monclient: shutdown 2024-03-25T19:35:52.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.046+0000 7f774107e640 1 -- 172.21.15.107:0/3983979133 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f772c001b90 con 0x7f773c0acd10 2024-03-25T19:35:52.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.046+0000 7f77442e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:52.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.046+0000 7f77442e4640 1 -- 172.21.15.107:0/3983979133 >> v1:172.21.15.107:6790/0 conn(0x7f773c0acd10 legacy=0x7f773c149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.046+0000 7f774107e640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-25T19:35:52.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.046+0000 7f77442e4640 1 -- 172.21.15.107:0/3983979133 shutdown_connections 2024-03-25T19:35:52.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.046+0000 7f77442e4640 1 -- 172.21.15.107:0/3983979133 >> 172.21.15.107:0/3983979133 conn(0x7f773c0b39c0 msgr2=0x7f773c0b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:52.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.047+0000 7f77442e4640 1 -- 172.21.15.107:0/3983979133 shutdown_connections 2024-03-25T19:35:52.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.047+0000 7f77442e4640 1 -- 172.21.15.107:0/3983979133 wait complete. 2024-03-25T19:35:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.048+0000 7f77442e4640 10 monclient: build_initial_monmap 2024-03-25T19:35:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.048+0000 7f77442e4640 10 monclient: monmap: 2024-03-25T19:35:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2024-03-25T19:35:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 0.000000 2024-03-25T19:35:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:created 0.000000 2024-03-25T19:35:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-25T19:35:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:35:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:35:52.049 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:52.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.048+0000 7f77442e4640 1 librados: starting msgr at 2024-03-25T19:35:52.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.048+0000 7f77442e4640 1 librados: starting objecter 2024-03-25T19:35:52.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.048+0000 7f77442e4640 1 Processor -- start 2024-03-25T19:35:52.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.048+0000 7f77442e4640 1 -- start start 2024-03-25T19:35:52.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.048+0000 7f77442e4640 1 librados: setting wanted keys 2024-03-25T19:35:52.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.048+0000 7f77442e4640 1 librados: calling monclient init 2024-03-25T19:35:52.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.048+0000 7f77442e4640 10 monclient: init 2024-03-25T19:35:52.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.049+0000 7f77442e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:52.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.049+0000 7f77442e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:52.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.049+0000 7f77442e4640 10 monclient(hunting): picked mon.noname-c con 0x7f773c0a9cb0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:52.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.049+0000 7f77442e4640 10 monclient(hunting): picked mon.noname-b con 0x7f773c0acd10 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:52.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.049+0000 7f77442e4640 10 monclient(hunting): picked mon.noname-a con 0x7f773c06df50 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:52.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.049+0000 7f77442e4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c1c78a0 con 0x7f773c06df50 2024-03-25T19:35:52.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.049+0000 7f77442e4640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c1c7a30 con 0x7f773c0acd10 2024-03-25T19:35:52.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.049+0000 7f77442e4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c1c4c60 con 0x7f773c0a9cb0 2024-03-25T19:35:52.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.049+0000 7f77442e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:52.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.049+0000 7f77442e4640 10 monclient(hunting): authenticate will time out at 4544.390430s 2024-03-25T19:35:52.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.050+0000 7f7742080640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f773c0a9cb0 0x7f773c1bc7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50726/0 (socket says 172.21.15.107:50726) 2024-03-25T19:35:52.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.050+0000 7f7742080640 1 -- 172.21.15.107:0/2538321933 learned_addr learned my addr 172.21.15.107:0/2538321933 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:35:52.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.051+0000 7f7732ffd640 10 client.?.objecter ms_handle_connect 0x7f773c06df50 2024-03-25T19:35:52.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.051+0000 7f7732ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:52.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.051+0000 7f7732ffd640 10 client.?.objecter ms_handle_connect 0x7f773c0a9cb0 2024-03-25T19:35:52.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.051+0000 7f7732ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:52.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.054+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474439413 0 0) 0x7f773c1c4c60 con 0x7f773c0a9cb0 2024-03-25T19:35:52.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.054+0000 7f7732ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:52.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.054+0000 7f7732ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:52.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.054+0000 7f7732ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:52.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.054+0000 7f7732ffd640 10 monclient(hunting): my global_id is 4163 2024-03-25T19:35:52.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.054+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7720004eb0 con 0x7f773c0a9cb0 2024-03-25T19:35:52.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.055+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821164375 0 0) 0x7f773c1c78a0 con 0x7f773c06df50 2024-03-25T19:35:52.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.055+0000 7f7732ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:52.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.055+0000 7f7732ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:52.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.055+0000 7f7732ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:52.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.055+0000 7f7732ffd640 10 monclient(hunting): my global_id is 4164 2024-03-25T19:35:52.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.055+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f773c1c4c60 con 0x7f773c06df50 2024-03-25T19:35:52.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.064+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1819286425 0 0) 0x7f773c1c4c60 con 0x7f773c06df50 2024-03-25T19:35:52.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.064+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f773c1c78a0 con 0x7f773c06df50 2024-03-25T19:35:52.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.065+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f772c0015f0 con 0x7f773c06df50 2024-03-25T19:35:52.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.065+0000 7f7732ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:52.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.065+0000 7f7732ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:52.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.065+0000 7f7732ffd640 10 monclient(hunting): dump: 2024-03-25T19:35:52.065 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-25T19:35:52.065 INFO:teuthology.orchestra.run.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:52.065 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:52.065 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:52.065 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:52.065 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:52.065 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:52.065 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:52.066 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:52.066 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:52.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.065+0000 7f7732ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:52.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.065+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3688412638 0 0) 0x7f7720004eb0 con 0x7f773c0a9cb0 2024-03-25T19:35:52.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.065+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f773c1c4c60 con 0x7f773c0a9cb0 2024-03-25T19:35:52.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.066+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f77380019f0 con 0x7f773c0a9cb0 2024-03-25T19:35:52.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.066+0000 7f7732ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:52.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.066+0000 7f7732ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:52.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.066+0000 7f7732ffd640 10 monclient(hunting): dump: 2024-03-25T19:35:52.067 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-25T19:35:52.067 INFO:teuthology.orchestra.run.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:52.067 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:52.067 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:52.067 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:52.067 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:52.067 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:52.067 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:52.067 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:52.067 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3285071030 0 0) 0x7f773c1c4c60 con 0x7f773c0a9cb0 2024-03-25T19:35:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 >> v1:172.21.15.195:6789/0 conn(0x7f773c0acd10 legacy=0x7f773c1d2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 >> v1:172.21.15.107:6789/0 conn(0x7f773c06df50 legacy=0x7f773c1d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f7732ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:52.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f7732ffd640 1 monclient: found mon.c 2024-03-25T19:35:52.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f7732ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:52.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f7732ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:52.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c1c4df0 con 0x7f773c0a9cb0 2024-03-25T19:35:52.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f7732ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:52.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f7732ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:52.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f7732ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:52.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.071+0000 7f77442e4640 5 monclient: authenticate success, global_id 4163 2024-03-25T19:35:52.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.072+0000 7f77442e4640 10 monclient: _renew_subs 2024-03-25T19:35:52.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.072+0000 7f77442e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:52.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.072+0000 7f77442e4640 1 -- 172.21.15.107:0/2538321933 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c1c5800 con 0x7f773c0a9cb0 2024-03-25T19:35:52.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.072+0000 7f77442e4640 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:52.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.072+0000 7f77442e4640 10 monclient: _renew_subs 2024-03-25T19:35:52.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.072+0000 7f77442e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:52.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.072+0000 7f77442e4640 1 -- 172.21.15.107:0/2538321933 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c1bced0 con 0x7f773c0a9cb0 2024-03-25T19:35:52.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.072+0000 7f77442e4640 1 librados: init done 2024-03-25T19:35:52.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.073+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7738003690 con 0x7f773c0a9cb0 2024-03-25T19:35:52.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.073+0000 7f7732ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:52.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.073+0000 7f774087d640 4 set_mon_vals no callback set 2024-03-25T19:35:52.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.074+0000 7f77442e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:52.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.074+0000 7f77442e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:52.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.074+0000 7f77442e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:52.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.074+0000 7f77442e4640 1 -- 172.21.15.107:0/2538321933 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7724005080 con 0x7f773c0a9cb0 2024-03-25T19:35:52.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.090+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f77380016d0 con 0x7f773c0a9cb0 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.090+0000 7f7732ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.090+0000 7f7732ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.090+0000 7f7732ffd640 10 monclient: dump: 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:52.091 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:35:52.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.091+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1006695550 0 0) 0x7f7738003b60 con 0x7f773c0a9cb0 2024-03-25T19:35:52.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.093+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f7738015720 con 0x7f773c0a9cb0 2024-03-25T19:35:52.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.093+0000 7f7732ffd640 10 client.4163.objecter ms_dispatch 0x7f773c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:52.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.093+0000 7f7732ffd640 3 client.4163.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:52.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.093+0000 7f7732ffd640 3 client.4163.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:52.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.094+0000 7f7732ffd640 20 client.4163.objecter dump_active .. 0 homeless 2024-03-25T19:35:52.098 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:52.099+0000 7f45c3e10040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:35:52.156 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:52.157+0000 7f45c3e10040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:35:52.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.277+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7738014df0 con 0x7f773c0a9cb0 2024-03-25T19:35:52.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.277+0000 7f7732ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:52.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.277+0000 7f7732ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:52.294 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-25T19:35:52.295+0000 7f45c3e10040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:35:52.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.400+0000 7f77442e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:52.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.400+0000 7f77442e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:52.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.400+0000 7f77442e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:52.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.400+0000 7f77442e4640 1 -- 172.21.15.107:0/2538321933 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7724005330 con 0x7f773c0a9cb0 2024-03-25T19:35:52.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.502+0000 7f7732ffd640 1 -- 172.21.15.107:0/2538321933 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f7738003ff0 con 0x7f773c0a9cb0 2024-03-25T19:35:52.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.502+0000 7f7732ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:52.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.502+0000 7f7732ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:52.502 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-25T19:35:52.502 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:52.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.503+0000 7f77442e4640 10 librados: watch_flush enter 2024-03-25T19:35:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.503+0000 7f77442e4640 10 librados: watch_flush exit 2024-03-25T19:35:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.503+0000 7f77442e4640 20 client.4163.objecter shutdown clearing up homeless session... 2024-03-25T19:35:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.503+0000 7f77442e4640 10 client.4163.objecter successfully canceled tick 2024-03-25T19:35:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.503+0000 7f77442e4640 10 monclient: shutdown 2024-03-25T19:35:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.503+0000 7f77442e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.503+0000 7f77442e4640 1 -- 172.21.15.107:0/2538321933 >> v1:172.21.15.107:6790/0 conn(0x7f773c0a9cb0 legacy=0x7f773c1bc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.503+0000 7f77442e4640 1 -- 172.21.15.107:0/2538321933 shutdown_connections 2024-03-25T19:35:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.503+0000 7f77442e4640 1 -- 172.21.15.107:0/2538321933 >> 172.21.15.107:0/2538321933 conn(0x7f773c0b39c0 msgr2=0x7f773c0ad160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.503+0000 7f77442e4640 1 -- 172.21.15.107:0/2538321933 shutdown_connections 2024-03-25T19:35:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.503+0000 7f77442e4640 1 -- 172.21.15.107:0/2538321933 wait complete. 2024-03-25T19:35:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-25T19:35:52.504+0000 7f77442e4640 1 librados: shutdown 2024-03-25T19:35:52.518 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:35:52.518 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:35:53.324 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:53.388 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:53.388+0000 7f1897a26640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:35:53.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.466+0000 7fc76b00f640 10 monclient: get_monmap_and_config 2024-03-25T19:35:53.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.466+0000 7fc76b00f640 10 monclient: build_initial_monmap 2024-03-25T19:35:53.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.466+0000 7fc76b00f640 10 monclient: monmap: 2024-03-25T19:35:53.466 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:35:53.466 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:53.466 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:35:53.467418+0000 2024-03-25T19:35:53.466 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:35:53.467418+0000 2024-03-25T19:35:53.467 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:53.467 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:53.467 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:53.467 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:35:53.467 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:35:53.467 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:35:53.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.467+0000 7fc76b00f640 1 Processor -- start 2024-03-25T19:35:53.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.467+0000 7fc76b00f640 1 -- start start 2024-03-25T19:35:53.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.467+0000 7fc76b00f640 10 monclient: init 2024-03-25T19:35:53.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.468+0000 7fc76b00f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:53.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.468+0000 7fc76b00f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:53.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.468+0000 7fc76b00f640 10 monclient(hunting): picked mon.noname-b con 0x7fc7640a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:53.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.468+0000 7fc76b00f640 10 monclient(hunting): picked mon.noname-c con 0x7fc76413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:53.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.468+0000 7fc76b00f640 10 monclient(hunting): picked mon.noname-a con 0x7fc7641430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:53.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.468+0000 7fc76b00f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc764142520 con 0x7fc7641430c0 2024-03-25T19:35:53.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.468+0000 7fc76b00f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7640b28a0 con 0x7fc76413f710 2024-03-25T19:35:53.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.468+0000 7fc76b00f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7640b2a30 con 0x7fc7640a98f0 2024-03-25T19:35:53.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.468+0000 7fc76b00f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:53.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.469+0000 7fc76b00f640 10 monclient(hunting): authenticate will time out at 4545.809808s 2024-03-25T19:35:53.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.469+0000 7fc7695ac640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc7641430c0 0x7fc76414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46870/0 (socket says 172.21.15.107:46870) 2024-03-25T19:35:53.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.469+0000 7fc7695ac640 1 -- 172.21.15.107:0/3481725436 learned_addr learned my addr 172.21.15.107:0/3481725436 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:35:53.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.472+0000 7fc75b7fe640 1 -- 172.21.15.107:0/3481725436 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984991692 0 0) 0x7fc7640b2a30 con 0x7fc7640a98f0 2024-03-25T19:35:53.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.472+0000 7fc75b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:53.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.473+0000 7fc75b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:53.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.473+0000 7fc75b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:53.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.473+0000 7fc75b7fe640 10 monclient(hunting): my global_id is 4178 2024-03-25T19:35:53.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.473+0000 7fc75b7fe640 1 -- 172.21.15.107:0/3481725436 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc74c003520 con 0x7fc7640a98f0 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.477+0000 7fc75b7fe640 1 -- 172.21.15.107:0/3481725436 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 568759943 0 0) 0x7fc74c003520 con 0x7fc7640a98f0 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.477+0000 7fc75b7fe640 1 -- 172.21.15.107:0/3481725436 >> v1:172.21.15.195:6789/0 conn(0x7fc76413f710 legacy=0x7fc7641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc75b7fe640 1 -- 172.21.15.107:0/3481725436 >> v1:172.21.15.107:6789/0 conn(0x7fc7641430c0 legacy=0x7fc76414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc75b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc75b7fe640 1 monclient: found mon.noname-b 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc75b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc75b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc75b7fe640 1 -- 172.21.15.107:0/3481725436 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7640b44c0 con 0x7fc7640a98f0 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc75b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc75b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc75b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:53.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc76b00f640 5 monclient: authenticate success, global_id 4178 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc76b00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc75b7fe640 1 -- 172.21.15.107:0/3481725436 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fc7600015c0 con 0x7fc7640a98f0 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.478+0000 7fc75b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.479+0000 7fc75b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.479+0000 7fc75b7fe640 10 monclient: dump: 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:53.480 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:53.481 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:53.481 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:35:53.481 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.479+0000 7fc76b00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:53.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.481+0000 7fc75b7fe640 1 -- 172.21.15.107:0/3481725436 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc760001890 con 0x7fc7640a98f0 2024-03-25T19:35:53.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.481+0000 7fc75b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:53.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.481+0000 7fc76b00f640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:53.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.481+0000 7fc76b00f640 4 set_mon_vals no callback set 2024-03-25T19:35:53.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.481+0000 7fc75b7fe640 1 -- 172.21.15.107:0/3481725436 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fc760001c20 con 0x7fc7640a98f0 2024-03-25T19:35:53.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.481+0000 7fc75b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:53.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.481+0000 7fc75b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:53.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.481+0000 7fc75b7fe640 10 monclient: dump: 2024-03-25T19:35:53.483 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.482+0000 7fc76b00f640 10 monclient: shutdown 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.482+0000 7fc76b00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:53.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.482+0000 7fc76b00f640 1 -- 172.21.15.107:0/3481725436 >> v1:172.21.15.107:6790/0 conn(0x7fc7640a98f0 legacy=0x7fc7640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:53.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.482+0000 7fc76b00f640 1 -- 172.21.15.107:0/3481725436 shutdown_connections 2024-03-25T19:35:53.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.482+0000 7fc76b00f640 1 -- 172.21.15.107:0/3481725436 >> 172.21.15.107:0/3481725436 conn(0x7fc7640b7ea0 msgr2=0x7fc7640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:53.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.483+0000 7fc76b00f640 1 -- 172.21.15.107:0/3481725436 shutdown_connections 2024-03-25T19:35:53.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.483+0000 7fc76b00f640 1 -- 172.21.15.107:0/3481725436 wait complete. 2024-03-25T19:35:53.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.484+0000 7fc76b00f640 10 monclient: build_initial_monmap 2024-03-25T19:35:53.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.484+0000 7fc76b00f640 10 monclient: monmap: 2024-03-25T19:35:53.488 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:35:53.488 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:53.488 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:35:53.488 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.484+0000 7fc76b00f640 1 librados: starting msgr at 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.484+0000 7fc76b00f640 1 librados: starting objecter 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.484+0000 7fc76b00f640 1 Processor -- start 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.485+0000 7fc76b00f640 1 -- start start 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.485+0000 7fc76b00f640 1 librados: setting wanted keys 2024-03-25T19:35:53.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.485+0000 7fc76b00f640 1 librados: calling monclient init 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.485+0000 7fc76b00f640 10 monclient: init 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.485+0000 7fc76b00f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.485+0000 7fc76b00f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.485+0000 7fc76b00f640 10 monclient(hunting): picked mon.noname-a con 0x7fc7640a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.485+0000 7fc76b00f640 10 monclient(hunting): picked mon.noname-b con 0x7fc76413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.486+0000 7fc76b00f640 10 monclient(hunting): picked mon.noname-c con 0x7fc7641430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.486+0000 7fc76b00f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc764139980 con 0x7fc7640a98f0 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.486+0000 7fc76b00f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7641d7810 con 0x7fc76413f710 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.486+0000 7fc76b00f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7641d8440 con 0x7fc7641430c0 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.486+0000 7fc76b00f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.486+0000 7fc76b00f640 10 monclient(hunting): authenticate will time out at 4545.826968s 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.486+0000 7fc768dab640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc7640a98f0 0x7fc7641d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46880/0 (socket says 172.21.15.107:46880) 2024-03-25T19:35:53.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.486+0000 7fc768dab640 1 -- 172.21.15.107:0/3273790391 learned_addr learned my addr 172.21.15.107:0/3273790391 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:35:53.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.487+0000 7fc7597fa640 10 client.?.objecter ms_handle_connect 0x7fc7640a98f0 2024-03-25T19:35:53.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.487+0000 7fc7597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:53.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.487+0000 7fc7597fa640 10 client.?.objecter ms_handle_connect 0x7fc7641430c0 2024-03-25T19:35:53.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.487+0000 7fc7597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:53.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.490+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709277743 0 0) 0x7fc7641d8440 con 0x7fc7641430c0 2024-03-25T19:35:53.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.490+0000 7fc7597fa640 10 monclient(hunting): _negotiate 2024-03-25T19:35:53.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.490+0000 7fc7597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:53.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.490+0000 7fc7597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:53.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.490+0000 7fc7597fa640 10 monclient(hunting): my global_id is 4181 2024-03-25T19:35:53.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.490+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc750004eb0 con 0x7fc7641430c0 2024-03-25T19:35:53.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.499+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1480733519 0 0) 0x7fc750004eb0 con 0x7fc7641430c0 2024-03-25T19:35:53.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.499+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7641d8440 con 0x7fc7641430c0 2024-03-25T19:35:53.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.499+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fc75c001b00 con 0x7fc7641430c0 2024-03-25T19:35:53.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.499+0000 7fc7597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:53.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.500+0000 7fc7597fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:53.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.500+0000 7fc7597fa640 10 monclient(hunting): dump: 2024-03-25T19:35:53.500 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:35:53.500 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:53.501 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:53.501 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:53.501 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:53.501 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:53.501 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:53.501 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:53.501 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:53.501 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:35:53.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.500+0000 7fc7597fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:53.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.505+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 389563036 0 0) 0x7fc7641d8440 con 0x7fc7641430c0 2024-03-25T19:35:53.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.505+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 >> v1:172.21.15.195:6789/0 conn(0x7fc76413f710 legacy=0x7fc7641d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:53.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 >> v1:172.21.15.107:6789/0 conn(0x7fc7640a98f0 legacy=0x7fc7641d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:53.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc7597fa640 10 monclient: _finish_hunting 0 2024-03-25T19:35:53.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc7597fa640 1 monclient: found mon.c 2024-03-25T19:35:53.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc7597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:53.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc7597fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7641d85d0 con 0x7fc7641430c0 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc7597fa640 10 monclient: _finish_auth 0 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc7597fa640 10 monclient: _check_auth_tickets 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc7597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc76b00f640 5 monclient: authenticate success, global_id 4181 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc76b00f640 10 monclient: _renew_subs 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc76b00f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc76b00f640 1 -- 172.21.15.107:0/3273790391 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7641d5000 con 0x7fc7641430c0 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc76b00f640 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc76b00f640 10 monclient: _renew_subs 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc76b00f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc76b00f640 1 -- 172.21.15.107:0/3273790391 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7641d43c0 con 0x7fc7641430c0 2024-03-25T19:35:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.506+0000 7fc76b00f640 1 librados: init done 2024-03-25T19:35:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.507+0000 7fc76b00f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.507+0000 7fc76b00f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.507+0000 7fc76b00f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.507+0000 7fc76b00f640 1 -- 172.21.15.107:0/3273790391 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7641367a0 con 0x7fc7641430c0 2024-03-25T19:35:53.694 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:35:53.695+0000 7f1897a26640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:35:53.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.990+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc75c0020b0 con 0x7fc7641430c0 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.990+0000 7fc7597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.990+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fc75c004d00 con 0x7fc7641430c0 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.990+0000 7fc7597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.990+0000 7fc7597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.990+0000 7fc7597fa640 10 monclient: dump: 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:35:53.990 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:35:53.991 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:35:53.991 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:35:53.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.990+0000 7fc75affd640 4 set_mon_vals no callback set 2024-03-25T19:35:53.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.991+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 266218084 0 0) 0x7fc75c004f20 con 0x7fc7641430c0 2024-03-25T19:35:53.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.992+0000 7fc768dab640 10 monclient: get_auth_request con 0x7fc75006db50 auth_method 0 2024-03-25T19:35:53.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:53.992+0000 7fc7597fa640 10 client.4181.objecter ms_handle_connect 0x7fc75006db50 2024-03-25T19:35:54.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.002+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fc75c0549e0 con 0x7fc7641430c0 2024-03-25T19:35:54.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.002+0000 7fc7597fa640 10 client.4181.objecter ms_dispatch 0x7fc764000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:54.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.002+0000 7fc7597fa640 3 client.4181.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:54.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.002+0000 7fc7597fa640 3 client.4181.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:54.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.003+0000 7fc7597fa640 20 client.4181.objecter dump_active .. 0 homeless 2024-03-25T19:35:54.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.003+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc75c0504f0 con 0x7fc7641430c0 2024-03-25T19:35:54.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.003+0000 7fc7597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:54.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.003+0000 7fc7597fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:54.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.365+0000 7fc76b00f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:54.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.365+0000 7fc76b00f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:54.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.365+0000 7fc76b00f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:35:54.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.365+0000 7fc76b00f640 1 -- 172.21.15.107:0/3273790391 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7641d7670 con 0x7fc7641430c0 2024-03-25T19:35:54.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.390+0000 7fc7597fa640 1 -- 172.21.15.107:0/3273790391 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fc7641d7670 con 0x7fc7641430c0 2024-03-25T19:35:54.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.390+0000 7fc7597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:54.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.390+0000 7fc7597fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:54.390 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:35:54.390 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:54.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.393+0000 7fc76b00f640 10 librados: watch_flush enter 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.393+0000 7fc76b00f640 10 librados: watch_flush exit 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.393+0000 7fc76b00f640 20 client.4181.objecter shutdown clearing up homeless session... 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.393+0000 7fc76b00f640 10 client.4181.objecter successfully canceled tick 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.393+0000 7fc76b00f640 1 -- 172.21.15.107:0/3273790391 >> v1:172.21.15.195:6800/2984408421 conn(0x7fc75006db50 legacy=0x7fc750070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.393+0000 7fc76b00f640 10 monclient: shutdown 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.393+0000 7fc76b00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.393+0000 7fc76b00f640 1 -- 172.21.15.107:0/3273790391 >> v1:172.21.15.107:6790/0 conn(0x7fc7641430c0 legacy=0x7fc7641381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.393+0000 7fc76b00f640 1 -- 172.21.15.107:0/3273790391 shutdown_connections 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.394+0000 7fc76b00f640 1 -- 172.21.15.107:0/3273790391 >> 172.21.15.107:0/3273790391 conn(0x7fc7640b7ea0 msgr2=0x7fc764074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.394+0000 7fc76b00f640 1 -- 172.21.15.107:0/3273790391 shutdown_connections 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.394+0000 7fc76b00f640 1 -- 172.21.15.107:0/3273790391 wait complete. 2024-03-25T19:35:54.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:35:54.394+0000 7fc76b00f640 1 librados: shutdown 2024-03-25T19:35:54.408 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:58.601 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:06.818 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:58.633 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:06.856 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:58.662 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:06.880 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:58.694 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:06.917 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:58.707 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:06.930 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:58.783 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:07.007 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:58.981 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:07.201 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.018 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:07.223 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.040 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:07.260 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.084 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:07.289 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.102 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:07.319 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.162 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:07.378 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.208 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-03-25T19:35:59.202+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:59.248 INFO:tasks.ceph.osd.5.smithi195.stderr:2024-03-25T19:35:59.243+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:59.252 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:07.471 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.314 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:07.532 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.333 INFO:tasks.ceph.osd.4.smithi195.stderr:2024-03-25T19:35:59.328+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:59.445 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:07.654 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.505 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:07.714 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.612 INFO:tasks.ceph.osd.6.smithi195.stderr:2024-03-25T19:35:59.606+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:59.631 INFO:tasks.ceph.osd.2.smithi107.stderr:2024-03-25T19:35:59.623+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:59.678 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:07.895 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.722 INFO:tasks.ceph.osd.1.smithi107.stderr:2024-03-25T19:35:59.716+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:59.747 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:07.970 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.797 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:08.021 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.886 INFO:tasks.ceph.osd.3.smithi107.stderr:2024-03-25T19:35:59.880+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:36:00.046 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:08.264 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.065 INFO:tasks.ceph.osd.0.smithi107.stderr:2024-03-25T19:36:00.059+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:36:00.116 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:08.340 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.119 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:08.324 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.125 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:08.345 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.175 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:08.399 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.183 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:08.399 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.360 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:08.578 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.372 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:08.590 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.437 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:08.660 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.494 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:08.699 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.502 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:08.722 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.524 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:08.732 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.542 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:08.766 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.551 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:08.767 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.624 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:08.842 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.702 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:08.925 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.738 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:08.956 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.847 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:09.071 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.851 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:09.056 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.856 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:09.076 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.899 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:09.108 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.901 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:09.118 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.906 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:09.122 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:00.966 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:09.189 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.049 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:09.267 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.135 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:09.359 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.142 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:09.347 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.156 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:09.377 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.178 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:09.395 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.196 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:09.412 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.215 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:01.228 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:09.437 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.231 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:09.454 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.301 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.302+0000 7faf1937c640 10 monclient: get_monmap_and_config 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.302+0000 7faf1937c640 10 monclient: build_initial_monmap 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.302+0000 7faf1937c640 10 monclient: monmap: 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:36:01.303572+0000 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:36:01.303572+0000 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:36:01.302 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:01.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.303+0000 7faf1937c640 1 Processor -- start 2024-03-25T19:36:01.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.303+0000 7faf1937c640 1 -- start start 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.303+0000 7faf1937c640 10 monclient: init 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.303+0000 7faf1937c640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.303+0000 7faf1937c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.303+0000 7faf1937c640 10 monclient(hunting): picked mon.noname-c con 0x7faf140b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.303+0000 7faf1937c640 10 monclient(hunting): picked mon.noname-b con 0x7faf140b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.303+0000 7faf1937c640 10 monclient(hunting): picked mon.noname-a con 0x7faf1414be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.303+0000 7faf1937c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf140af940 con 0x7faf1414be90 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.303+0000 7faf1937c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf140b0a70 con 0x7faf140b4d60 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.304+0000 7faf1937c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf140b0d50 con 0x7faf140b1730 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.304+0000 7faf1937c640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.304+0000 7faf1937c640 10 monclient(hunting): authenticate will time out at 4553.644769s 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.304+0000 7faf137fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7faf1414be90 0x7faf1414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34816/0 (socket says 172.21.15.107:34816) 2024-03-25T19:36:01.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.304+0000 7faf137fe640 1 -- 172.21.15.107:0/599545941 learned_addr learned my addr 172.21.15.107:0/599545941 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:01.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223799042 0 0) 0x7faf140af940 con 0x7faf1414be90 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): my global_id is 4200 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf00003520 con 0x7faf1414be90 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059479678 0 0) 0x7faf140b0a70 con 0x7faf140b4d60 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): my global_id is 4192 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf140af940 con 0x7faf140b4d60 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823169514 0 0) 0x7faf140b0d50 con 0x7faf140b1730 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:36:01.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.306+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf140b0a70 con 0x7faf140b1730 2024-03-25T19:36:01.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3250819145 0 0) 0x7faf00003520 con 0x7faf1414be90 2024-03-25T19:36:01.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 >> v1:172.21.15.107:6790/0 conn(0x7faf140b1730 legacy=0x7faf140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:01.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 >> v1:172.21.15.195:6789/0 conn(0x7faf140b4d60 legacy=0x7faf1413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:01.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:01.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 1 monclient: found mon.noname-a 2024-03-25T19:36:01.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:01.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf140b1030 con 0x7faf1414be90 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faf0c001a10 con 0x7faf1414be90 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf127fc640 10 monclient: dump: 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:01.310 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:01.311 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:01.311 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:01.311 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:01.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf1937c640 5 monclient: authenticate success, global_id 4200 2024-03-25T19:36:01.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.309+0000 7faf1937c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:01.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.311+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf0c001b70 con 0x7faf1414be90 2024-03-25T19:36:01.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.311+0000 7faf127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:01.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.311+0000 7faf127fc640 1 -- 172.21.15.107:0/599545941 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faf0c001f80 con 0x7faf1414be90 2024-03-25T19:36:01.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.311+0000 7faf127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:01.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.311+0000 7faf127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.311+0000 7faf127fc640 10 monclient: dump: 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.311+0000 7faf1937c640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:01.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.311+0000 7faf1937c640 4 set_mon_vals no callback set 2024-03-25T19:36:01.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.312+0000 7faf1937c640 10 monclient: shutdown 2024-03-25T19:36:01.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.312+0000 7faf1937c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:01.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.312+0000 7faf1937c640 1 -- 172.21.15.107:0/599545941 >> v1:172.21.15.107:6789/0 conn(0x7faf1414be90 legacy=0x7faf1414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.312+0000 7faf1937c640 1 -- 172.21.15.107:0/599545941 shutdown_connections 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.312+0000 7faf1937c640 1 -- 172.21.15.107:0/599545941 >> 172.21.15.107:0/599545941 conn(0x7faf140617a0 msgr2=0x7faf14061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.312+0000 7faf1937c640 1 -- 172.21.15.107:0/599545941 shutdown_connections 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.312+0000 7faf1937c640 1 -- 172.21.15.107:0/599545941 wait complete. 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.313+0000 7faf1937c640 10 monclient: build_initial_monmap 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.313+0000 7faf1937c640 10 monclient: monmap: 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:36:01.315 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.313+0000 7faf1937c640 1 librados: starting msgr at 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.313+0000 7faf1937c640 1 librados: starting objecter 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.313+0000 7faf1937c640 1 Processor -- start 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.313+0000 7faf1937c640 1 -- start start 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.313+0000 7faf1937c640 1 librados: setting wanted keys 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.313+0000 7faf1937c640 1 librados: calling monclient init 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.313+0000 7faf1937c640 10 monclient: init 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf1937c640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf1937c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf1937c640 10 monclient(hunting): picked mon.noname-b con 0x7faf140b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf1937c640 10 monclient(hunting): picked mon.noname-c con 0x7faf140b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf1937c640 10 monclient(hunting): picked mon.noname-a con 0x7faf1414be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf1937c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf14141950 con 0x7faf1414be90 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf1937c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf141d7940 con 0x7faf140b1730 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf1937c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf141d8570 con 0x7faf140b4d60 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf1937c640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf1937c640 10 monclient(hunting): authenticate will time out at 4553.655104s 2024-03-25T19:36:01.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf137fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7faf1414be90 0x7faf141e2fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34828/0 (socket says 172.21.15.107:34828) 2024-03-25T19:36:01.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.314+0000 7faf137fe640 1 -- 172.21.15.107:0/733483509 learned_addr learned my addr 172.21.15.107:0/733483509 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:01.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.315+0000 7faf0b7fe640 10 client.?.objecter ms_handle_connect 0x7faf1414be90 2024-03-25T19:36:01.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.315+0000 7faf0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:01.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.315+0000 7faf0b7fe640 10 client.?.objecter ms_handle_connect 0x7faf140b1730 2024-03-25T19:36:01.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.315+0000 7faf0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:01.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.315+0000 7faf0b7fe640 10 client.?.objecter ms_handle_connect 0x7faf140b4d60 2024-03-25T19:36:01.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.315+0000 7faf0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:01.317 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:09.533 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985280635 0 0) 0x7faf14141950 con 0x7faf1414be90 2024-03-25T19:36:01.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:01.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:01.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:01.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 10 monclient(hunting): my global_id is 4203 2024-03-25T19:36:01.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef4004eb0 con 0x7faf1414be90 2024-03-25T19:36:01.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833524658 0 0) 0x7faf141d8570 con 0x7faf140b4d60 2024-03-25T19:36:01.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:01.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:01.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:01.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:36:01.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.317+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf14141950 con 0x7faf140b4d60 2024-03-25T19:36:01.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.318+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405501729 0 0) 0x7faf141d7940 con 0x7faf140b1730 2024-03-25T19:36:01.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.318+0000 7faf0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:01.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.318+0000 7faf0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:01.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.318+0000 7faf0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:01.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.318+0000 7faf0b7fe640 10 monclient(hunting): my global_id is 4195 2024-03-25T19:36:01.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.318+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf141d8570 con 0x7faf140b1730 2024-03-25T19:36:01.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4038465689 0 0) 0x7faf14141950 con 0x7faf140b4d60 2024-03-25T19:36:01.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf141d7940 con 0x7faf140b4d60 2024-03-25T19:36:01.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 296389600 0 0) 0x7faef4004eb0 con 0x7faf1414be90 2024-03-25T19:36:01.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf14141950 con 0x7faf1414be90 2024-03-25T19:36:01.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faf0c001ec0 con 0x7faf1414be90 2024-03-25T19:36:01.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faefc001930 con 0x7faf140b4d60 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:01.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448425237 0 0) 0x7faf141d8570 con 0x7faf140b1730 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faef4004eb0 con 0x7faf140b1730 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faf040018f0 con 0x7faf140b1730 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.322+0000 7faf0b7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:01.324 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:01.325 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:01.325 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.325 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.325 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:01.325 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:01.325 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:01.325 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:01.325 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:01.325 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:01.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 522005660 0 0) 0x7faef4004eb0 con 0x7faf140b1730 2024-03-25T19:36:01.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 >> v1:172.21.15.107:6790/0 conn(0x7faf140b4d60 legacy=0x7faf14142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:01.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 >> v1:172.21.15.107:6789/0 conn(0x7faf1414be90 legacy=0x7faf141e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf0b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf0b7fe640 1 monclient: found mon.b 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf0b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf14138840 con 0x7faf140b1730 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf0b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf0b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf1937c640 5 monclient: authenticate success, global_id 4195 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf1937c640 10 monclient: _renew_subs 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf1937c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf1937c640 1 -- 172.21.15.107:0/733483509 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf14142f70 con 0x7faf140b1730 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf1937c640 10 client.4195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf1937c640 10 monclient: _renew_subs 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf1937c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf1937c640 1 -- 172.21.15.107:0/733483509 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf141432c0 con 0x7faf140b1730 2024-03-25T19:36:01.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.326+0000 7faf1937c640 1 librados: init done 2024-03-25T19:36:01.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.327+0000 7faf1937c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:01.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.327+0000 7faf1937c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:01.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.327+0000 7faf1937c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:01.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.327+0000 7faf1937c640 1 -- 172.21.15.107:0/733483509 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf141d7ad0 con 0x7faf140b1730 2024-03-25T19:36:01.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.340+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf040035f0 con 0x7faf140b1730 2024-03-25T19:36:01.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.341+0000 7faf0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:01.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.341+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faf04003900 con 0x7faf140b1730 2024-03-25T19:36:01.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.341+0000 7faf0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:01.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.341+0000 7faf0b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:01.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.341+0000 7faf0b7fe640 10 monclient: dump: 2024-03-25T19:36:01.340 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:01.341 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:01.341 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.341 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:01.341 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:01.341 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:01.341 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:01.341 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:01.341 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:01.341 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:01.341 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.341+0000 7faf11ffb640 4 set_mon_vals no callback set 2024-03-25T19:36:01.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.341+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7faf040198d0 con 0x7faf140b1730 2024-03-25T19:36:01.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.342+0000 7faf0bfff640 10 monclient: get_auth_request con 0x7faef406e110 auth_method 0 2024-03-25T19:36:01.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.342+0000 7faf0b7fe640 10 client.4195.objecter ms_handle_connect 0x7faef406e110 2024-03-25T19:36:01.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.343+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7faf04053580 con 0x7faf140b1730 2024-03-25T19:36:01.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.343+0000 7faf0b7fe640 10 client.4195.objecter ms_dispatch 0x7faf14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:01.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.343+0000 7faf0b7fe640 3 client.4195.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:01.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.343+0000 7faf0b7fe640 3 client.4195.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:01.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.343+0000 7faf0b7fe640 20 client.4195.objecter dump_active .. 0 homeless 2024-03-25T19:36:01.427 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:09.651 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.432 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:09.637 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.454 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:09.674 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.454 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:09.671 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.484 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:09.700 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.500 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:09.708 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.507 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:09.730 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.570+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf0404eec0 con 0x7faf140b1730 2024-03-25T19:36:01.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.570+0000 7faf0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:01.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.570+0000 7faf0b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:01.590 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:09.808 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.723 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:09.947 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.725 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:09.930 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.727 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:09.945 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.744+0000 7faf1937c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:01.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.744+0000 7faf1937c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:01.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.744+0000 7faf1937c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:01.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.744+0000 7faf1937c640 1 -- 172.21.15.107:0/733483509 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf141389d0 con 0x7faf140b1730 2024-03-25T19:36:01.752 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:09.972 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.772 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:09.980 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.777 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:09.994 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.782 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:10.005 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.794+0000 7faf0b7fe640 1 -- 172.21.15.107:0/733483509 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7faf04050fc0 con 0x7faf140b1730 2024-03-25T19:36:01.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.794+0000 7faf0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:01.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.794+0000 7faf0b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:01.794 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:36:01.794 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 10 librados: watch_flush enter 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 10 librados: watch_flush exit 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 20 client.4195.objecter shutdown clearing up homeless session... 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 10 client.4195.objecter successfully canceled tick 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 1 -- 172.21.15.107:0/733483509 >> v1:172.21.15.195:6800/2984408421 conn(0x7faef406e110 legacy=0x7faef40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 10 monclient: shutdown 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 1 -- 172.21.15.107:0/733483509 >> v1:172.21.15.195:6789/0 conn(0x7faf140b1730 legacy=0x7faf141d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 1 -- 172.21.15.107:0/733483509 shutdown_connections 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 1 -- 172.21.15.107:0/733483509 >> 172.21.15.107:0/733483509 conn(0x7faf140617a0 msgr2=0x7faf140b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 1 -- 172.21.15.107:0/733483509 shutdown_connections 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 1 -- 172.21.15.107:0/733483509 wait complete. 2024-03-25T19:36:01.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:01.796+0000 7faf1937c640 1 librados: shutdown 2024-03-25T19:36:01.805 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:01.862 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:10.080 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:01.992 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:10.209 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.013 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:10.218 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.024 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:10.248 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.045 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:10.254 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.049 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:10.270 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.050 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:10.273 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.079 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:10.296 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.130 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:10.348 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.302 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:10.507 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.323 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:10.531 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.326 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:10.549 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.347 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:10.567 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.374 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:10.590 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.396 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:10.615 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.483 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:10.701 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.523 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:10.746 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.536 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:10.754 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.576 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:10.799 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.594 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:10.802 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.598 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:10.803 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.664 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:10.884 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.668 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:10.886 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.673 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:10.889 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.846 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:11.070 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.858 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:11.066 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:02.900 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:11.124 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:03.114 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:11.319 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:03.143 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:11.361 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:03.169 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:11.374 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:03.198 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:11.416 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:03.227 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:11.443 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:03.257 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:11.478 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:03.282 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:11.498 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:03.312 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:11.532 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:03.380 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:11.589 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:03.437 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:11.646 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:07.692 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:15.909 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:07.729 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:15.953 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:08.166 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:16.390 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:08.366 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:16.587 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:08.521 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:16.725 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:08.612 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:08.644 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:16.862 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:08.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.708+0000 7fd52353a640 10 monclient: get_monmap_and_config 2024-03-25T19:36:08.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.708+0000 7fd52353a640 10 monclient: build_initial_monmap 2024-03-25T19:36:08.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.708+0000 7fd52353a640 10 monclient: monmap: 2024-03-25T19:36:08.709 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:08.709 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:08.709 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:36:08.710305+0000 2024-03-25T19:36:08.709 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:36:08.710305+0000 2024-03-25T19:36:08.709 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:08.709 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:08.709 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:08.709 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:36:08.709 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:36:08.710 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.708+0000 7fd52353a640 1 Processor -- start 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.709+0000 7fd52353a640 1 -- start start 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.709+0000 7fd52353a640 10 monclient: init 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.709+0000 7fd52353a640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.709+0000 7fd52353a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.710+0000 7fd52353a640 10 monclient(hunting): picked mon.noname-c con 0x7fd51c0b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.710+0000 7fd52353a640 10 monclient(hunting): picked mon.noname-a con 0x7fd51c0b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.710+0000 7fd52353a640 10 monclient(hunting): picked mon.noname-b con 0x7fd51c14be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.710+0000 7fd52353a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c0af940 con 0x7fd51c0b1730 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.710+0000 7fd52353a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c0b0a70 con 0x7fd51c0b4d60 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.710+0000 7fd52353a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c0b0d50 con 0x7fd51c14be90 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.710+0000 7fd52353a640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:08.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.710+0000 7fd52353a640 10 monclient(hunting): authenticate will time out at 4561.052268s 2024-03-25T19:36:08.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.710+0000 7fd5212d6640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd51c0b4d60 0x7fd51c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:37638/0 (socket says 172.21.15.107:37638) 2024-03-25T19:36:08.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.710+0000 7fd5212d6640 1 -- 172.21.15.107:0/2847380026 learned_addr learned my addr 172.21.15.107:0/2847380026 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:08.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.710+0000 7fd520ad5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd51c0b1730 0x7fd51c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34844/0 (socket says 172.21.15.107:34844) 2024-03-25T19:36:08.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417997224 0 0) 0x7fd51c0b0d50 con 0x7fd51c14be90 2024-03-25T19:36:08.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:08.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:08.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:08.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:36:08.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd508003520 con 0x7fd51c14be90 2024-03-25T19:36:08.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508001572 0 0) 0x7fd51c0af940 con 0x7fd51c0b1730 2024-03-25T19:36:08.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:08.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:08.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:08.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:36:08.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c0b0d50 con 0x7fd51c0b1730 2024-03-25T19:36:08.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454358190 0 0) 0x7fd51c0b0a70 con 0x7fd51c0b4d60 2024-03-25T19:36:08.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:08.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:08.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:08.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 10 monclient(hunting): my global_id is 4198 2024-03-25T19:36:08.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.714+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c0af940 con 0x7fd51c0b4d60 2024-03-25T19:36:08.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 915284759 0 0) 0x7fd508003520 con 0x7fd51c14be90 2024-03-25T19:36:08.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 >> v1:172.21.15.195:6789/0 conn(0x7fd51c0b4d60 legacy=0x7fd51c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:08.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 >> v1:172.21.15.107:6789/0 conn(0x7fd51c0b1730 legacy=0x7fd51c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:08.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:08.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 1 monclient: found mon.noname-b 2024-03-25T19:36:08.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:08.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:36:08.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c0b1030 con 0x7fd51c14be90 2024-03-25T19:36:08.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 10 monclient: _finish_auth 0 2024-03-25T19:36:08.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:08.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:08.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd514001a10 con 0x7fd51c14be90 2024-03-25T19:36:08.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:08.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:08.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd513fff640 10 monclient: dump: 2024-03-25T19:36:08.720 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:08.720 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:08.720 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.720 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.720 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:08.720 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:08.721 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:08.721 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:08.721 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:08.721 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:08.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd52353a640 5 monclient: authenticate success, global_id 4214 2024-03-25T19:36:08.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.718+0000 7fd52353a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:08.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.720+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd514001b70 con 0x7fd51c14be90 2024-03-25T19:36:08.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.720+0000 7fd513fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:08.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.720+0000 7fd513fff640 1 -- 172.21.15.107:0/2847380026 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd514001f80 con 0x7fd51c14be90 2024-03-25T19:36:08.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.720+0000 7fd513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:08.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.720+0000 7fd513fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:08.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.720+0000 7fd513fff640 10 monclient: dump: 2024-03-25T19:36:08.725 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:08.725 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:08.725 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.725 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.725 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.720+0000 7fd52353a640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.720+0000 7fd52353a640 4 set_mon_vals no callback set 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.721+0000 7fd52353a640 10 monclient: shutdown 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.721+0000 7fd52353a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.721+0000 7fd52353a640 1 -- 172.21.15.107:0/2847380026 >> v1:172.21.15.107:6790/0 conn(0x7fd51c14be90 legacy=0x7fd51c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.721+0000 7fd52353a640 1 -- 172.21.15.107:0/2847380026 shutdown_connections 2024-03-25T19:36:08.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.721+0000 7fd52353a640 1 -- 172.21.15.107:0/2847380026 >> 172.21.15.107:0/2847380026 conn(0x7fd51c0617a0 msgr2=0x7fd51c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.721+0000 7fd52353a640 1 -- 172.21.15.107:0/2847380026 shutdown_connections 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.721+0000 7fd52353a640 1 -- 172.21.15.107:0/2847380026 wait complete. 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.722+0000 7fd52353a640 10 monclient: build_initial_monmap 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.722+0000 7fd52353a640 10 monclient: monmap: 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:08.727 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:36:08.728 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:36:08.728 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:08.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.722+0000 7fd52353a640 1 librados: starting msgr at 2024-03-25T19:36:08.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.722+0000 7fd52353a640 1 librados: starting objecter 2024-03-25T19:36:08.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.722+0000 7fd52353a640 1 Processor -- start 2024-03-25T19:36:08.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.722+0000 7fd52353a640 1 -- start start 2024-03-25T19:36:08.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.722+0000 7fd52353a640 1 librados: setting wanted keys 2024-03-25T19:36:08.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.722+0000 7fd52353a640 1 librados: calling monclient init 2024-03-25T19:36:08.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.722+0000 7fd52353a640 10 monclient: init 2024-03-25T19:36:08.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd52353a640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:08.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd52353a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:08.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd52353a640 10 monclient(hunting): picked mon.noname-b con 0x7fd51c0b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:08.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd52353a640 10 monclient(hunting): picked mon.noname-a con 0x7fd51c0b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:08.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd52353a640 10 monclient(hunting): picked mon.noname-c con 0x7fd51c14be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:08.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd52353a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c140f90 con 0x7fd51c0b4d60 2024-03-25T19:36:08.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd52353a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c1327c0 con 0x7fd51c0b1730 2024-03-25T19:36:08.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd52353a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c138290 con 0x7fd51c14be90 2024-03-25T19:36:08.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd52353a640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:08.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd52353a640 10 monclient(hunting): authenticate will time out at 4561.065084s 2024-03-25T19:36:08.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd520ad5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd51c0b4d60 0x7fd51c1daca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34850/0 (socket says 172.21.15.107:34850) 2024-03-25T19:36:08.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.723+0000 7fd520ad5640 1 -- 172.21.15.107:0/253418424 learned_addr learned my addr 172.21.15.107:0/253418424 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:08.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.724+0000 7fd511ffb640 10 client.?.objecter ms_handle_connect 0x7fd51c0b4d60 2024-03-25T19:36:08.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.724+0000 7fd511ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:08.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.724+0000 7fd511ffb640 10 client.?.objecter ms_handle_connect 0x7fd51c14be90 2024-03-25T19:36:08.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.724+0000 7fd511ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:08.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.724+0000 7fd511ffb640 10 client.?.objecter ms_handle_connect 0x7fd51c0b1730 2024-03-25T19:36:08.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.724+0000 7fd511ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:08.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259372145 0 0) 0x7fd51c140f90 con 0x7fd51c0b4d60 2024-03-25T19:36:08.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:08.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:08.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:08.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:36:08.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd500004eb0 con 0x7fd51c0b4d60 2024-03-25T19:36:08.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244389632 0 0) 0x7fd51c138290 con 0x7fd51c14be90 2024-03-25T19:36:08.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:08.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:08.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:08.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c140f90 con 0x7fd51c14be90 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834428287 0 0) 0x7fd51c1327c0 con 0x7fd51c0b1730 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 10 monclient(hunting): my global_id is 4201 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.726+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c138290 con 0x7fd51c0b1730 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.731+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1443171937 0 0) 0x7fd500004eb0 con 0x7fd51c0b4d60 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.731+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd51c1327c0 con 0x7fd51c0b4d60 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.731+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd50c0019f0 con 0x7fd51c0b4d60 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.731+0000 7fd511ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:08.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.731+0000 7fd511ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.731+0000 7fd511ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.731+0000 7fd511ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.732+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1521797408 0 0) 0x7fd51c140f90 con 0x7fd51c14be90 2024-03-25T19:36:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.732+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd500004eb0 con 0x7fd51c14be90 2024-03-25T19:36:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.733+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd514001e80 con 0x7fd51c14be90 2024-03-25T19:36:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.733+0000 7fd511ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.733+0000 7fd511ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.733+0000 7fd511ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:08.737 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:08.737 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:08.737 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.737 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.737 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:08.738 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:08.738 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:08.738 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:08.738 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:08.738 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:08.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.733+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2733604096 0 0) 0x7fd51c138290 con 0x7fd51c0b1730 2024-03-25T19:36:08.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.734+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd51c140f90 con 0x7fd51c0b1730 2024-03-25T19:36:08.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.734+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd518001830 con 0x7fd51c0b1730 2024-03-25T19:36:08.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.734+0000 7fd511ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:08.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.734+0000 7fd511ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.734+0000 7fd511ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3668026943 0 0) 0x7fd51c1327c0 con 0x7fd51c0b4d60 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 >> v1:172.21.15.107:6790/0 conn(0x7fd51c14be90 legacy=0x7fd51c1db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 >> v1:172.21.15.195:6789/0 conn(0x7fd51c0b1730 legacy=0x7fd51c1320b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:08.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd511ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd511ffb640 1 monclient: found mon.a 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd511ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd511ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c140380 con 0x7fd51c0b4d60 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd511ffb640 10 monclient: _finish_auth 0 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd511ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd511ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd52353a640 5 monclient: authenticate success, global_id 4209 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd52353a640 10 monclient: _renew_subs 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd52353a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd52353a640 1 -- 172.21.15.107:0/253418424 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51c140510 con 0x7fd51c0b4d60 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd52353a640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd52353a640 10 monclient: _renew_subs 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd52353a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd52353a640 1 -- 172.21.15.107:0/253418424 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51c144f90 con 0x7fd51c0b4d60 2024-03-25T19:36:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.736+0000 7fd52353a640 1 librados: init done 2024-03-25T19:36:08.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.738+0000 7fd52353a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:08.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.738+0000 7fd52353a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:08.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.738+0000 7fd52353a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:36:08.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.738+0000 7fd52353a640 1 -- 172.21.15.107:0/253418424 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd51c0627b0 con 0x7fd51c0b4d60 2024-03-25T19:36:08.742 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:16.956 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:08.742 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:16.950 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:08.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.754+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd50c0047e0 con 0x7fd51c0b4d60 2024-03-25T19:36:08.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.754+0000 7fd511ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:08.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.754+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd50c004af0 con 0x7fd51c0b4d60 2024-03-25T19:36:08.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.754+0000 7fd511ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:08.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.754+0000 7fd511ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:08.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.754+0000 7fd511ffb640 10 monclient: dump: 2024-03-25T19:36:08.756 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:08.756 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:08.756 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.756 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:08.757 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:08.757 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:08.757 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:08.757 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:08.757 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:08.757 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:08.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.754+0000 7fd5137fe640 4 set_mon_vals no callback set 2024-03-25T19:36:08.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.755+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fd50c01ab00 con 0x7fd51c0b4d60 2024-03-25T19:36:08.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.755+0000 7fd5212d6640 10 monclient: get_auth_request con 0x7fd50006e240 auth_method 0 2024-03-25T19:36:08.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.756+0000 7fd511ffb640 10 client.4209.objecter ms_handle_connect 0x7fd50006e240 2024-03-25T19:36:08.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.757+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fd50c054840 con 0x7fd51c0b4d60 2024-03-25T19:36:08.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.757+0000 7fd511ffb640 10 client.4209.objecter ms_dispatch 0x7fd51c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:08.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.757+0000 7fd511ffb640 3 client.4209.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:08.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.757+0000 7fd511ffb640 3 client.4209.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:08.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:08.757+0000 7fd511ffb640 20 client.4209.objecter dump_active .. 0 homeless 2024-03-25T19:36:09.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.122+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd50c050180 con 0x7fd51c0b4d60 2024-03-25T19:36:09.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.122+0000 7fd511ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:09.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.122+0000 7fd511ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:09.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.317+0000 7fd52353a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:09.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.317+0000 7fd52353a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:09.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.317+0000 7fd52353a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:36:09.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.317+0000 7fd52353a640 1 -- 172.21.15.107:0/253418424 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd51c069f40 con 0x7fd51c0b4d60 2024-03-25T19:36:09.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.399+0000 7fd511ffb640 1 -- 172.21.15.107:0/253418424 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fd50c052280 con 0x7fd51c0b4d60 2024-03-25T19:36:09.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.399+0000 7fd511ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:09.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.399+0000 7fd511ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:09.400 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:36:09.400 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:09.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.402+0000 7fd52353a640 10 librados: watch_flush enter 2024-03-25T19:36:09.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.402+0000 7fd52353a640 10 librados: watch_flush exit 2024-03-25T19:36:09.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.402+0000 7fd52353a640 20 client.4209.objecter shutdown clearing up homeless session... 2024-03-25T19:36:09.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.402+0000 7fd52353a640 10 client.4209.objecter successfully canceled tick 2024-03-25T19:36:09.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.402+0000 7fd52353a640 1 -- 172.21.15.107:0/253418424 >> v1:172.21.15.195:6800/2984408421 conn(0x7fd50006e240 legacy=0x7fd500070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:09.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.402+0000 7fd52353a640 10 monclient: shutdown 2024-03-25T19:36:09.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.402+0000 7fd52353a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:09.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.402+0000 7fd52353a640 1 -- 172.21.15.107:0/253418424 >> v1:172.21.15.107:6789/0 conn(0x7fd51c0b4d60 legacy=0x7fd51c1daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:09.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.402+0000 7fd52353a640 1 -- 172.21.15.107:0/253418424 shutdown_connections 2024-03-25T19:36:09.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.402+0000 7fd52353a640 1 -- 172.21.15.107:0/253418424 >> 172.21.15.107:0/253418424 conn(0x7fd51c0617a0 msgr2=0x7fd51c0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:09.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.403+0000 7fd52353a640 1 -- 172.21.15.107:0/253418424 shutdown_connections 2024-03-25T19:36:09.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.403+0000 7fd52353a640 1 -- 172.21.15.107:0/253418424 wait complete. 2024-03-25T19:36:09.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:09.404+0000 7fd52353a640 1 librados: shutdown 2024-03-25T19:36:09.414 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:13.393 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:21.610 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:13.397 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:21.614 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:13.400 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:21.624 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:13.404 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:21.628 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:13.770 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:21.993 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:13.773 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:21.997 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:13.980 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:22.201 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:13.986 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:22.206 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:14.527 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:22.735 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:14.530 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:22.739 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:14.602 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:22.819 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:14.605 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:22.823 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:14.631 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:22.847 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:14.635 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:22.851 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:14.691 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:22.896 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:14.695 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:22.900 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:16.221 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:16.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.315+0000 7f15cdfc8640 10 monclient: get_monmap_and_config 2024-03-25T19:36:16.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.315+0000 7f15cdfc8640 10 monclient: build_initial_monmap 2024-03-25T19:36:16.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.315+0000 7f15cdfc8640 10 monclient: monmap: 2024-03-25T19:36:16.315 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:16.315 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:16.315 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:36:16.316610+0000 2024-03-25T19:36:16.315 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:36:16.316610+0000 2024-03-25T19:36:16.315 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:16.315 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:16.315 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:16.316 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:36:16.316 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:36:16.316 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:16.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.316+0000 7f15cdfc8640 1 Processor -- start 2024-03-25T19:36:16.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.316+0000 7f15cdfc8640 1 -- start start 2024-03-25T19:36:16.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.316+0000 7f15cdfc8640 10 monclient: init 2024-03-25T19:36:16.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15cdfc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:16.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15cdfc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:16.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15cdfc8640 10 monclient(hunting): picked mon.noname-b con 0x7f15c80b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:16.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15cdfc8640 10 monclient(hunting): picked mon.noname-a con 0x7f15c80b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:16.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15cdfc8640 10 monclient(hunting): picked mon.noname-c con 0x7f15c814be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:16.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15cdfc8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c80af940 con 0x7f15c80b1730 2024-03-25T19:36:16.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15cdfc8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c80b0a70 con 0x7f15c814be90 2024-03-25T19:36:16.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15cdfc8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c80b0d50 con 0x7f15c80b4d60 2024-03-25T19:36:16.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15cdfc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:16.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15cdfc8640 10 monclient(hunting): authenticate will time out at 4568.658174s 2024-03-25T19:36:16.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15c6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f15c80b1730 0x7f15c80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34244/0 (socket says 172.21.15.107:34244) 2024-03-25T19:36:16.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.317+0000 7f15c6ffd640 1 -- 172.21.15.107:0/913332380 learned_addr learned my addr 172.21.15.107:0/913332380 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:16.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.320+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888729258 0 0) 0x7f15c80b0a70 con 0x7f15c814be90 2024-03-25T19:36:16.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.320+0000 7f15c67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:16.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.320+0000 7f15c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:16.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.320+0000 7f15c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:16.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.320+0000 7f15c67fc640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:36:16.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.320+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b0003520 con 0x7f15c814be90 2024-03-25T19:36:16.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236123232 0 0) 0x7f15c80af940 con 0x7f15c80b1730 2024-03-25T19:36:16.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:16.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:16.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:16.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:36:16.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c80b0a70 con 0x7f15c80b1730 2024-03-25T19:36:16.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242868108 0 0) 0x7f15c80b0d50 con 0x7f15c80b4d60 2024-03-25T19:36:16.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:16.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:16.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:16.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 10 monclient(hunting): my global_id is 4220 2024-03-25T19:36:16.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.321+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c80af940 con 0x7f15c80b4d60 2024-03-25T19:36:16.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716029398 0 0) 0x7f15b0003520 con 0x7f15c814be90 2024-03-25T19:36:16.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 >> v1:172.21.15.107:6790/0 conn(0x7f15c80b4d60 legacy=0x7f15c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:16.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 >> v1:172.21.15.107:6789/0 conn(0x7f15c80b1730 legacy=0x7f15c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:16.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15c67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:16.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15c67fc640 1 monclient: found mon.noname-c 2024-03-25T19:36:16.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:16.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15c67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:36:16.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c80b1030 con 0x7f15c814be90 2024-03-25T19:36:16.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15c67fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:16.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15c67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:16.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:16.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15cdfc8640 5 monclient: authenticate success, global_id 4204 2024-03-25T19:36:16.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.324+0000 7f15cdfc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f15c00019d0 con 0x7f15c814be90 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15c67fc640 10 monclient: dump: 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:16.326 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15c0001c20 con 0x7f15c814be90 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15cdfc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15cdfc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15cdfc8640 4 set_mon_vals no callback set 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15c67fc640 1 -- 172.21.15.107:0/913332380 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f15c0001ff0 con 0x7f15c814be90 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.326+0000 7f15c67fc640 10 monclient: dump: 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:16.327 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:16.328 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:16.328 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:16.328 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:16.328 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:16.328 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:16.328 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:16.328 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:16.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.327+0000 7f15cdfc8640 10 monclient: shutdown 2024-03-25T19:36:16.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.327+0000 7f15cdfc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.327+0000 7f15cdfc8640 1 -- 172.21.15.107:0/913332380 >> v1:172.21.15.195:6789/0 conn(0x7f15c814be90 legacy=0x7f15c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.327+0000 7f15cdfc8640 1 -- 172.21.15.107:0/913332380 shutdown_connections 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.327+0000 7f15cdfc8640 1 -- 172.21.15.107:0/913332380 >> 172.21.15.107:0/913332380 conn(0x7f15c80617a0 msgr2=0x7f15c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.327+0000 7f15cdfc8640 1 -- 172.21.15.107:0/913332380 shutdown_connections 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.328+0000 7f15cdfc8640 1 -- 172.21.15.107:0/913332380 wait complete. 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.328+0000 7f15cdfc8640 10 monclient: build_initial_monmap 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.328+0000 7f15cdfc8640 10 monclient: monmap: 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:16.332 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.329+0000 7f15cdfc8640 1 librados: starting msgr at 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.329+0000 7f15cdfc8640 1 librados: starting objecter 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.329+0000 7f15cdfc8640 1 Processor -- start 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.329+0000 7f15cdfc8640 1 -- start start 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.329+0000 7f15cdfc8640 1 librados: setting wanted keys 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.329+0000 7f15cdfc8640 1 librados: calling monclient init 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.329+0000 7f15cdfc8640 10 monclient: init 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15cdfc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15cdfc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15cdfc8640 10 monclient(hunting): picked mon.noname-c con 0x7f15c80b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:16.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15cdfc8640 10 monclient(hunting): picked mon.noname-b con 0x7f15c80b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15cdfc8640 10 monclient(hunting): picked mon.noname-a con 0x7f15c814be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15cdfc8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c81380f0 con 0x7f15c814be90 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15cdfc8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c8138320 con 0x7f15c80b4d60 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15cdfc8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c8135fa0 con 0x7f15c80b1730 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15cdfc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15cdfc8640 10 monclient(hunting): authenticate will time out at 4568.671085s 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15c7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f15c814be90 0x7f15c81e3070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34252/0 (socket says 172.21.15.107:34252) 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.330+0000 7f15c7fff640 1 -- 172.21.15.107:0/1578568271 learned_addr learned my addr 172.21.15.107:0/1578568271 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.331+0000 7f15affff640 10 client.?.objecter ms_handle_connect 0x7f15c814be90 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.331+0000 7f15affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.331+0000 7f15affff640 10 client.?.objecter ms_handle_connect 0x7f15c80b4d60 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.331+0000 7f15affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.331+0000 7f15affff640 10 client.?.objecter ms_handle_connect 0x7f15c80b1730 2024-03-25T19:36:16.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.331+0000 7f15affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:16.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884303173 0 0) 0x7f15c8135fa0 con 0x7f15c80b1730 2024-03-25T19:36:16.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:16.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 10 monclient(hunting): my global_id is 4223 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b4004eb0 con 0x7f15c80b1730 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209650083 0 0) 0x7f15c81380f0 con 0x7f15c814be90 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.333+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c8135fa0 con 0x7f15c814be90 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.336+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887346628 0 0) 0x7f15c8138320 con 0x7f15c80b4d60 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.336+0000 7f15affff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.336+0000 7f15affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.336+0000 7f15affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:16.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.336+0000 7f15affff640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:36:16.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.336+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c81380f0 con 0x7f15c80b4d60 2024-03-25T19:36:16.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 447471180 0 0) 0x7f15c8135fa0 con 0x7f15c814be90 2024-03-25T19:36:16.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15c8138320 con 0x7f15c814be90 2024-03-25T19:36:16.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f15c0001f30 con 0x7f15c814be90 2024-03-25T19:36:16.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:16.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:16.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 10 monclient(hunting): dump: 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 360701165 0 0) 0x7f15b4004eb0 con 0x7f15c80b1730 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15c8135fa0 con 0x7f15c80b1730 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f15b80018f0 con 0x7f15c80b1730 2024-03-25T19:36:16.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.338+0000 7f15affff640 10 monclient(hunting): dump: 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:16.341 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:16.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.340+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 766019166 0 0) 0x7f15c8138320 con 0x7f15c814be90 2024-03-25T19:36:16.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.340+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 >> v1:172.21.15.107:6790/0 conn(0x7f15c80b1730 legacy=0x7f15c812fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:16.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 >> v1:172.21.15.195:6789/0 conn(0x7f15c80b4d60 legacy=0x7f15c8136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:16.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15affff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:16.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15affff640 1 monclient: found mon.a 2024-03-25T19:36:16.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:16.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15affff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:36:16.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c8136130 con 0x7f15c814be90 2024-03-25T19:36:16.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15affff640 10 monclient: _finish_auth 0 2024-03-25T19:36:16.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15affff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15cdfc8640 5 monclient: authenticate success, global_id 4215 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15cdfc8640 10 monclient: _renew_subs 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15cdfc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15cdfc8640 1 -- 172.21.15.107:0/1578568271 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c81db950 con 0x7f15c814be90 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15cdfc8640 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15cdfc8640 10 monclient: _renew_subs 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15cdfc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15cdfc8640 1 -- 172.21.15.107:0/1578568271 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c8130500 con 0x7f15c814be90 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.341+0000 7f15cdfc8640 1 librados: init done 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.342+0000 7f15cdfc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.342+0000 7f15cdfc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.342+0000 7f15cdfc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:36:16.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.342+0000 7f15cdfc8640 1 -- 172.21.15.107:0/1578568271 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c8130850 con 0x7f15c814be90 2024-03-25T19:36:16.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.609+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15c00020d0 con 0x7f15c814be90 2024-03-25T19:36:16.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.609+0000 7f15affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:16.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.609+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f15c0003dc0 con 0x7f15c814be90 2024-03-25T19:36:16.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.609+0000 7f15affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:16.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.609+0000 7f15affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:16.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.609+0000 7f15affff640 10 monclient: dump: 2024-03-25T19:36:16.609 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:16.609 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:16.610 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:16.610 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:16.610 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:16.610 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:16.610 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:16.610 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:16.610 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:16.610 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:16.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.609+0000 7f15c5ffb640 4 set_mon_vals no callback set 2024-03-25T19:36:16.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.610+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f15c0003fe0 con 0x7f15c814be90 2024-03-25T19:36:16.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.611+0000 7f15c77fe640 10 monclient: get_auth_request con 0x7f15b406e110 auth_method 0 2024-03-25T19:36:16.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.611+0000 7f15affff640 10 client.4215.objecter ms_handle_connect 0x7f15b406e110 2024-03-25T19:36:16.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.615+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f15c0054a00 con 0x7f15c814be90 2024-03-25T19:36:16.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.615+0000 7f15affff640 10 client.4215.objecter ms_dispatch 0x7f15c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:16.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.615+0000 7f15affff640 3 client.4215.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:16.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.615+0000 7f15affff640 3 client.4215.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:16.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.615+0000 7f15affff640 20 client.4215.objecter dump_active .. 0 homeless 2024-03-25T19:36:16.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.615+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15c0050510 con 0x7f15c814be90 2024-03-25T19:36:16.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.615+0000 7f15affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:16.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.615+0000 7f15affff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:16.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.950+0000 7f15cdfc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:16.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.950+0000 7f15cdfc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:16.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.950+0000 7f15cdfc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:36:16.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.950+0000 7f15cdfc8640 1 -- 172.21.15.107:0/1578568271 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15c80627b0 con 0x7f15c814be90 2024-03-25T19:36:16.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.967+0000 7f15affff640 1 -- 172.21.15.107:0/1578568271 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f15c0052440 con 0x7f15c814be90 2024-03-25T19:36:16.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.967+0000 7f15affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:16.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.967+0000 7f15affff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:16.967 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:36:16.967 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:16.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.969+0000 7f15cdfc8640 10 librados: watch_flush enter 2024-03-25T19:36:16.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.969+0000 7f15cdfc8640 10 librados: watch_flush exit 2024-03-25T19:36:16.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.969+0000 7f15cdfc8640 20 client.4215.objecter shutdown clearing up homeless session... 2024-03-25T19:36:16.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.969+0000 7f15cdfc8640 10 client.4215.objecter successfully canceled tick 2024-03-25T19:36:16.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.969+0000 7f15cdfc8640 1 -- 172.21.15.107:0/1578568271 >> v1:172.21.15.195:6800/2984408421 conn(0x7f15b406e110 legacy=0x7f15b40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:16.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.969+0000 7f15cdfc8640 10 monclient: shutdown 2024-03-25T19:36:16.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.969+0000 7f15cdfc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:16.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.969+0000 7f15cdfc8640 1 -- 172.21.15.107:0/1578568271 >> v1:172.21.15.107:6789/0 conn(0x7f15c814be90 legacy=0x7f15c81e3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:16.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.969+0000 7f15cdfc8640 1 -- 172.21.15.107:0/1578568271 shutdown_connections 2024-03-25T19:36:16.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.969+0000 7f15cdfc8640 1 -- 172.21.15.107:0/1578568271 >> 172.21.15.107:0/1578568271 conn(0x7f15c80617a0 msgr2=0x7f15c813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:16.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.970+0000 7f15cdfc8640 1 -- 172.21.15.107:0/1578568271 shutdown_connections 2024-03-25T19:36:16.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.970+0000 7f15cdfc8640 1 -- 172.21.15.107:0/1578568271 wait complete. 2024-03-25T19:36:16.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:16.970+0000 7f15cdfc8640 1 librados: shutdown 2024-03-25T19:36:16.979 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:17.634 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:00:25.858 36062== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:17.730 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:00:25.947 36064== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:18.078 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:00:26.302 36061== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:18.419 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:00:26.639 36063== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:18.657 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:00:26.865 35655== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:18.971 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:00:27.188 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:19.191 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:00:27.407 35654== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:19.263 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:00:27.467 35656== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:23.787 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:23.879 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.879+0000 7f4c5dc84640 10 monclient: get_monmap_and_config 2024-03-25T19:36:23.879 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.879+0000 7f4c5dc84640 10 monclient: build_initial_monmap 2024-03-25T19:36:23.879 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.879+0000 7f4c5dc84640 10 monclient: monmap: 2024-03-25T19:36:23.879 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:23.879 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:23.879 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:36:23.880165+0000 2024-03-25T19:36:23.879 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:36:23.880165+0000 2024-03-25T19:36:23.880 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:23.880 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:23.880 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:23.880 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:36:23.880 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:36:23.880 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:23.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 1 Processor -- start 2024-03-25T19:36:23.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 1 -- start start 2024-03-25T19:36:23.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 10 monclient: init 2024-03-25T19:36:23.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:23.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:23.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 10 monclient(hunting): picked mon.noname-b con 0x7f4c580b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:23.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 10 monclient(hunting): picked mon.noname-a con 0x7f4c580b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:23.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 10 monclient(hunting): picked mon.noname-c con 0x7f4c5814be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:23.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c580af940 con 0x7f4c580b1730 2024-03-25T19:36:23.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c580b0a70 con 0x7f4c5814be90 2024-03-25T19:36:23.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c580b0d50 con 0x7f4c580b4d60 2024-03-25T19:36:23.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:23.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.880+0000 7f4c5dc84640 10 monclient(hunting): authenticate will time out at 4576.221534s 2024-03-25T19:36:23.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.881+0000 7f4c56ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4c580b1730 0x7f4c580b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58822/0 (socket says 172.21.15.107:58822) 2024-03-25T19:36:23.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.881+0000 7f4c56ffd640 1 -- 172.21.15.107:0/1352979716 learned_addr learned my addr 172.21.15.107:0/1352979716 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:23.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.883+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655864680 0 0) 0x7f4c580af940 con 0x7f4c580b1730 2024-03-25T19:36:23.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.883+0000 7f4c567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:23.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.883+0000 7f4c567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:23.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.883+0000 7f4c567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:23.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.883+0000 7f4c567fc640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:36:23.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.883+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c40003520 con 0x7f4c580b1730 2024-03-25T19:36:23.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.885+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989577117 0 0) 0x7f4c580b0d50 con 0x7f4c580b4d60 2024-03-25T19:36:23.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.885+0000 7f4c567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:23.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.885+0000 7f4c567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:23.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.885+0000 7f4c567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:23.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.885+0000 7f4c567fc640 10 monclient(hunting): my global_id is 4226 2024-03-25T19:36:23.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.885+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c580af940 con 0x7f4c580b4d60 2024-03-25T19:36:23.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.886+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673442253 0 0) 0x7f4c580b0a70 con 0x7f4c5814be90 2024-03-25T19:36:23.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.886+0000 7f4c567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:23.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.886+0000 7f4c567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:23.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.886+0000 7f4c567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:23.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:36:23.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c580b0d50 con 0x7f4c5814be90 2024-03-25T19:36:23.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 649506803 0 0) 0x7f4c40003520 con 0x7f4c580b1730 2024-03-25T19:36:23.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 >> v1:172.21.15.107:6790/0 conn(0x7f4c580b4d60 legacy=0x7f4c5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:23.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 >> v1:172.21.15.195:6789/0 conn(0x7f4c5814be90 legacy=0x7f4c5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:23.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:23.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 1 monclient: found mon.noname-a 2024-03-25T19:36:23.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c580b1030 con 0x7f4c580b1730 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4c4c001690 con 0x7f4c580b1730 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.887+0000 7f4c5dc84640 5 monclient: authenticate success, global_id 4218 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.888+0000 7f4c5dc84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.888+0000 7f4c567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.888+0000 7f4c567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.888+0000 7f4c567fc640 10 monclient: dump: 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:23.888 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:23.889 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:23.889 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:23.889 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:23.889 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:23.889 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:23.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.888+0000 7f4c5dc84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:23.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.890+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c4c0018b0 con 0x7f4c580b1730 2024-03-25T19:36:23.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.890+0000 7f4c567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:23.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.890+0000 7f4c5dc84640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:23.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.890+0000 7f4c5dc84640 4 set_mon_vals no callback set 2024-03-25T19:36:23.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.891+0000 7f4c5dc84640 10 monclient: shutdown 2024-03-25T19:36:23.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.891+0000 7f4c567fc640 1 -- 172.21.15.107:0/1352979716 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4c4c001c40 con 0x7f4c580b1730 2024-03-25T19:36:23.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.891+0000 7f4c567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:23.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.891+0000 7f4c567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:23.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.891+0000 7f4c567fc640 10 monclient: dump: 2024-03-25T19:36:23.892 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:23.892 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:23.892 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:23.892 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:23.892 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:23.892 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:23.893 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:23.893 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:23.893 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:23.893 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:23.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.892+0000 7f4c5dc84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:23.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.892+0000 7f4c5dc84640 1 -- 172.21.15.107:0/1352979716 >> v1:172.21.15.107:6789/0 conn(0x7f4c580b1730 legacy=0x7f4c580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:23.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.892+0000 7f4c5dc84640 1 -- 172.21.15.107:0/1352979716 shutdown_connections 2024-03-25T19:36:23.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.892+0000 7f4c5dc84640 1 -- 172.21.15.107:0/1352979716 >> 172.21.15.107:0/1352979716 conn(0x7f4c580617a0 msgr2=0x7f4c58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:23.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.892+0000 7f4c5dc84640 1 -- 172.21.15.107:0/1352979716 shutdown_connections 2024-03-25T19:36:23.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.892+0000 7f4c5dc84640 1 -- 172.21.15.107:0/1352979716 wait complete. 2024-03-25T19:36:23.894 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.893+0000 7f4c5dc84640 10 monclient: build_initial_monmap 2024-03-25T19:36:23.894 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.893+0000 7f4c5dc84640 10 monclient: monmap: 2024-03-25T19:36:23.894 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.894+0000 7f4c5dc84640 1 librados: starting msgr at 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.894+0000 7f4c5dc84640 1 librados: starting objecter 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.894+0000 7f4c5dc84640 1 Processor -- start 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.894+0000 7f4c5dc84640 1 -- start start 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.895+0000 7f4c5dc84640 1 librados: setting wanted keys 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.895+0000 7f4c5dc84640 1 librados: calling monclient init 2024-03-25T19:36:23.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.895+0000 7f4c5dc84640 10 monclient: init 2024-03-25T19:36:23.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.896+0000 7f4c5dc84640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:23.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.896+0000 7f4c5dc84640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:23.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.896+0000 7f4c5dc84640 10 monclient(hunting): picked mon.noname-c con 0x7f4c580b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:23.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.896+0000 7f4c5dc84640 10 monclient(hunting): picked mon.noname-b con 0x7f4c580b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:23.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.896+0000 7f4c5dc84640 10 monclient(hunting): picked mon.noname-a con 0x7f4c5814be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:23.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.896+0000 7f4c5dc84640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c58140e50 con 0x7f4c5814be90 2024-03-25T19:36:23.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.896+0000 7f4c5dc84640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c58140930 con 0x7f4c580b4d60 2024-03-25T19:36:23.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.896+0000 7f4c5dc84640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c58138760 con 0x7f4c580b1730 2024-03-25T19:36:23.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.897+0000 7f4c56ffd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4c580b4d60 0x7f4c581dab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:34296/0 (socket says 172.21.15.107:34296) 2024-03-25T19:36:23.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.897+0000 7f4c56ffd640 1 -- 172.21.15.107:0/2945097441 learned_addr learned my addr 172.21.15.107:0/2945097441 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:23.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.897+0000 7f4c5dc84640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:23.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.897+0000 7f4c5dc84640 10 monclient(hunting): authenticate will time out at 4576.238049s 2024-03-25T19:36:23.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.897+0000 7f4c3ffff640 10 client.?.objecter ms_handle_connect 0x7f4c580b4d60 2024-03-25T19:36:23.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.897+0000 7f4c3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:23.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.897+0000 7f4c3ffff640 10 client.?.objecter ms_handle_connect 0x7f4c5814be90 2024-03-25T19:36:23.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.897+0000 7f4c3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:23.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.899+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551854903 0 0) 0x7f4c58140930 con 0x7f4c580b4d60 2024-03-25T19:36:23.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.900+0000 7f4c3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:23.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.900+0000 7f4c3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:23.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.900+0000 7f4c3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:23.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.900+0000 7f4c3ffff640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:36:23.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.900+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c44004eb0 con 0x7f4c580b4d60 2024-03-25T19:36:23.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.900+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128979715 0 0) 0x7f4c58140e50 con 0x7f4c5814be90 2024-03-25T19:36:23.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.900+0000 7f4c3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:23.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.900+0000 7f4c3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:23.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.900+0000 7f4c3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:23.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.900+0000 7f4c3ffff640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:36:23.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.900+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c58140930 con 0x7f4c5814be90 2024-03-25T19:36:23.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.906+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3941908887 0 0) 0x7f4c44004eb0 con 0x7f4c580b4d60 2024-03-25T19:36:23.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.906+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c58140e50 con 0x7f4c580b4d60 2024-03-25T19:36:23.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.906+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4c4c001690 con 0x7f4c580b4d60 2024-03-25T19:36:23.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.906+0000 7f4c3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:23.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.906+0000 7f4c3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.906+0000 7f4c3ffff640 10 monclient(hunting): dump: 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:23.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.906+0000 7f4c3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:23.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.908+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3694273925 0 0) 0x7f4c58140930 con 0x7f4c5814be90 2024-03-25T19:36:23.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.908+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c44004eb0 con 0x7f4c5814be90 2024-03-25T19:36:23.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.908+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4c50001bd0 con 0x7f4c5814be90 2024-03-25T19:36:23.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.908+0000 7f4c3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:23.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.908+0000 7f4c3ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:23.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.908+0000 7f4c3ffff640 10 monclient(hunting): dump: 2024-03-25T19:36:23.908 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:23.908 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:23.908 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:23.908 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:23.909 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:23.909 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:23.909 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:23.909 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:23.909 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:23.909 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:23.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.911+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2633361688 0 0) 0x7f4c58140e50 con 0x7f4c580b4d60 2024-03-25T19:36:23.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.911+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 >> v1:172.21.15.107:6790/0 conn(0x7f4c580b1730 legacy=0x7f4c58131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:23.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.911+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 >> v1:172.21.15.107:6789/0 conn(0x7f4c5814be90 legacy=0x7f4c581e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:23.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.911+0000 7f4c3ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:23.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.911+0000 7f4c3ffff640 1 monclient: found mon.b 2024-03-25T19:36:23.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.911+0000 7f4c3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:23.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.911+0000 7f4c3ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:23.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.911+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c581388f0 con 0x7f4c580b4d60 2024-03-25T19:36:23.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.911+0000 7f4c3ffff640 10 monclient: _finish_auth 0 2024-03-25T19:36:23.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.911+0000 7f4c3ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:23.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.911+0000 7f4c3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:23.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.912+0000 7f4c5dc84640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:36:23.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.912+0000 7f4c5dc84640 10 monclient: _renew_subs 2024-03-25T19:36:23.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.912+0000 7f4c5dc84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:23.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.912+0000 7f4c5dc84640 1 -- 172.21.15.107:0/2945097441 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c58137e90 con 0x7f4c580b4d60 2024-03-25T19:36:23.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.912+0000 7f4c5dc84640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:23.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.912+0000 7f4c5dc84640 10 monclient: _renew_subs 2024-03-25T19:36:23.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.912+0000 7f4c5dc84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:23.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.912+0000 7f4c5dc84640 1 -- 172.21.15.107:0/2945097441 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c581384a0 con 0x7f4c580b4d60 2024-03-25T19:36:23.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.912+0000 7f4c5dc84640 1 librados: init done 2024-03-25T19:36:23.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.913+0000 7f4c5dc84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:23.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.913+0000 7f4c5dc84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:23.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.913+0000 7f4c5dc84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:23.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.913+0000 7f4c5dc84640 1 -- 172.21.15.107:0/2945097441 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c58140b70 con 0x7f4c580b4d60 2024-03-25T19:36:23.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.936+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c4c0011f0 con 0x7f4c580b4d60 2024-03-25T19:36:23.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.936+0000 7f4c3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:23.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.936+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4c4c004a90 con 0x7f4c580b4d60 2024-03-25T19:36:23.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.936+0000 7f4c3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:23.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.936+0000 7f4c3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:23.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.936+0000 7f4c3ffff640 10 monclient: dump: 2024-03-25T19:36:23.937 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:23.937 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:23.937 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:23.937 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:23.937 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:23.937 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:23.937 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:23.937 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:23.937 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:23.937 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:23.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.936+0000 7f4c55ffb640 4 set_mon_vals no callback set 2024-03-25T19:36:23.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.937+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f4c4c01aae0 con 0x7f4c580b4d60 2024-03-25T19:36:23.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.938+0000 7f4c577fe640 10 monclient: get_auth_request con 0x7f4c4406ddf0 auth_method 0 2024-03-25T19:36:23.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.938+0000 7f4c3ffff640 10 client.4213.objecter ms_handle_connect 0x7f4c4406ddf0 2024-03-25T19:36:23.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.939+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f4c4c0547c0 con 0x7f4c580b4d60 2024-03-25T19:36:23.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.939+0000 7f4c3ffff640 10 client.4213.objecter ms_dispatch 0x7f4c58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:23.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.939+0000 7f4c3ffff640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:23.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.939+0000 7f4c3ffff640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:23.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:23.939+0000 7f4c3ffff640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-25T19:36:24.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.191+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c4c050100 con 0x7f4c580b4d60 2024-03-25T19:36:24.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.191+0000 7f4c3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:24.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.191+0000 7f4c3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:24.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.545+0000 7f4c5dc84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:24.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.545+0000 7f4c5dc84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:24.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.545+0000 7f4c5dc84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:24.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.545+0000 7f4c5dc84640 1 -- 172.21.15.107:0/2945097441 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c581401c0 con 0x7f4c580b4d60 2024-03-25T19:36:24.564 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:36:24.564 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:24.564 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.564+0000 7f4c3ffff640 1 -- 172.21.15.107:0/2945097441 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f4c4c052200 con 0x7f4c580b4d60 2024-03-25T19:36:24.564 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.564+0000 7f4c3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:24.564 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.564+0000 7f4c3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:24.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.566+0000 7f4c5dc84640 10 librados: watch_flush enter 2024-03-25T19:36:24.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 10 librados: watch_flush exit 2024-03-25T19:36:24.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-25T19:36:24.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 10 client.4213.objecter successfully canceled tick 2024-03-25T19:36:24.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 1 -- 172.21.15.107:0/2945097441 >> v1:172.21.15.195:6800/2984408421 conn(0x7f4c4406ddf0 legacy=0x7f4c440702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:24.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 10 monclient: shutdown 2024-03-25T19:36:24.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:24.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 1 -- 172.21.15.107:0/2945097441 >> v1:172.21.15.195:6789/0 conn(0x7f4c580b4d60 legacy=0x7f4c581dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:24.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 1 -- 172.21.15.107:0/2945097441 shutdown_connections 2024-03-25T19:36:24.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 1 -- 172.21.15.107:0/2945097441 >> 172.21.15.107:0/2945097441 conn(0x7f4c580617a0 msgr2=0x7f4c5813c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:24.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 1 -- 172.21.15.107:0/2945097441 shutdown_connections 2024-03-25T19:36:24.568 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 1 -- 172.21.15.107:0/2945097441 wait complete. 2024-03-25T19:36:24.568 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:24.567+0000 7f4c5dc84640 1 librados: shutdown 2024-03-25T19:36:24.577 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:26.539 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-03-25T19:36:26.536+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:36:26.746 INFO:tasks.ceph.osd.5.smithi195.stderr:2024-03-25T19:36:26.742+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:36:26.851 INFO:tasks.ceph.osd.4.smithi195.stderr:2024-03-25T19:36:26.848+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:36:27.090 INFO:tasks.ceph.osd.0.smithi107.stderr:2024-03-25T19:36:27.088+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:36:27.292 INFO:tasks.ceph.osd.6.smithi195.stderr:2024-03-25T19:36:27.288+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:36:27.483 INFO:tasks.ceph.osd.1.smithi107.stderr:2024-03-25T19:36:27.481+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:36:27.680 INFO:tasks.ceph.osd.2.smithi107.stderr:2024-03-25T19:36:27.673+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:36:27.769 INFO:tasks.ceph.osd.3.smithi107.stderr:2024-03-25T19:36:27.766+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:36:31.384 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:31.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.500+0000 7f242afd6640 10 monclient: get_monmap_and_config 2024-03-25T19:36:31.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.500+0000 7f242afd6640 10 monclient: build_initial_monmap 2024-03-25T19:36:31.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.500+0000 7f242afd6640 10 monclient: monmap: 2024-03-25T19:36:31.500 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:31.500 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:31.500 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:36:31.501381+0000 2024-03-25T19:36:31.500 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:36:31.501381+0000 2024-03-25T19:36:31.501 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:31.501 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:31.501 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:31.501 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:36:31.501 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:36:31.501 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:31.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.501+0000 7f242afd6640 1 Processor -- start 2024-03-25T19:36:31.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.501+0000 7f242afd6640 1 -- start start 2024-03-25T19:36:31.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.501+0000 7f242afd6640 10 monclient: init 2024-03-25T19:36:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.502+0000 7f242afd6640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.502+0000 7f242afd6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.502+0000 7f242afd6640 10 monclient(hunting): picked mon.noname-c con 0x7f24240b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.502+0000 7f242afd6640 10 monclient(hunting): picked mon.noname-a con 0x7f24240b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.502+0000 7f242afd6640 10 monclient(hunting): picked mon.noname-b con 0x7f242414be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.502+0000 7f242afd6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24240af940 con 0x7f24240b1730 2024-03-25T19:36:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.502+0000 7f242afd6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24240b0a70 con 0x7f24240b4d60 2024-03-25T19:36:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.502+0000 7f242afd6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24240b0d50 con 0x7f242414be90 2024-03-25T19:36:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.502+0000 7f242afd6640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.502+0000 7f242afd6640 10 monclient(hunting): authenticate will time out at 4583.843028s 2024-03-25T19:36:31.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.510+0000 7f2423fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f24240b1730 0x7f24240b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43032/0 (socket says 172.21.15.107:43032) 2024-03-25T19:36:31.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.510+0000 7f2423fff640 1 -- 172.21.15.107:0/1097024877 learned_addr learned my addr 172.21.15.107:0/1097024877 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.523+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095801523 0 0) 0x7f24240b0a70 con 0x7f24240b4d60 2024-03-25T19:36:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.523+0000 7f24237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.523+0000 7f24237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.523+0000 7f24237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.523+0000 7f24237fe640 10 monclient(hunting): my global_id is 4240 2024-03-25T19:36:31.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.523+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2414003520 con 0x7f24240b4d60 2024-03-25T19:36:31.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.528+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236359099 0 0) 0x7f24240af940 con 0x7f24240b1730 2024-03-25T19:36:31.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.528+0000 7f24237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:31.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.528+0000 7f24237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:31.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.528+0000 7f24237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:31.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.528+0000 7f24237fe640 10 monclient(hunting): my global_id is 4248 2024-03-25T19:36:31.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.528+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24240b0a70 con 0x7f24240b1730 2024-03-25T19:36:31.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.532+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443782737 0 0) 0x7f24240b0d50 con 0x7f242414be90 2024-03-25T19:36:31.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.532+0000 7f24237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:31.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.532+0000 7f24237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:31.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.532+0000 7f24237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:31.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.532+0000 7f24237fe640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:36:31.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.532+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24240af940 con 0x7f242414be90 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2296181832 0 0) 0x7f24240b0a70 con 0x7f24240b1730 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 >> v1:172.21.15.107:6790/0 conn(0x7f242414be90 legacy=0x7f242414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 >> v1:172.21.15.195:6789/0 conn(0x7f24240b4d60 legacy=0x7f242413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 1 monclient: found mon.noname-a 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24240b1030 con 0x7f24240b1730 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2410001670 con 0x7f24240b1730 2024-03-25T19:36:31.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f24237fe640 10 monclient: dump: 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f242afd6640 5 monclient: authenticate success, global_id 4248 2024-03-25T19:36:31.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.536+0000 7f242afd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:31.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.540+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24100017d0 con 0x7f24240b1730 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.540+0000 7f24237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.540+0000 7f24237fe640 1 -- 172.21.15.107:0/1097024877 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2410001be0 con 0x7f24240b1730 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.540+0000 7f242afd6640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.540+0000 7f242afd6640 4 set_mon_vals no callback set 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.540+0000 7f24237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.540+0000 7f24237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.540+0000 7f24237fe640 10 monclient: dump: 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:31.540 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:31.541 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:31.541 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:31.541 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:31.541 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:31.541 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:31.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.541+0000 7f242afd6640 10 monclient: shutdown 2024-03-25T19:36:31.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.541+0000 7f242afd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:31.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.541+0000 7f242afd6640 1 -- 172.21.15.107:0/1097024877 >> v1:172.21.15.107:6789/0 conn(0x7f24240b1730 legacy=0x7f24240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:31.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.541+0000 7f242afd6640 1 -- 172.21.15.107:0/1097024877 shutdown_connections 2024-03-25T19:36:31.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.541+0000 7f242afd6640 1 -- 172.21.15.107:0/1097024877 >> 172.21.15.107:0/1097024877 conn(0x7f24240617a0 msgr2=0x7f2424061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.541+0000 7f242afd6640 1 -- 172.21.15.107:0/1097024877 shutdown_connections 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.541+0000 7f242afd6640 1 -- 172.21.15.107:0/1097024877 wait complete. 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.542+0000 7f242afd6640 10 monclient: build_initial_monmap 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.542+0000 7f242afd6640 10 monclient: monmap: 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:36:31.543 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:31.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.542+0000 7f242afd6640 1 librados: starting msgr at 2024-03-25T19:36:31.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.542+0000 7f242afd6640 1 librados: starting objecter 2024-03-25T19:36:31.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.542+0000 7f242afd6640 1 Processor -- start 2024-03-25T19:36:31.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.542+0000 7f242afd6640 1 -- start start 2024-03-25T19:36:31.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.542+0000 7f242afd6640 1 librados: setting wanted keys 2024-03-25T19:36:31.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.542+0000 7f242afd6640 1 librados: calling monclient init 2024-03-25T19:36:31.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.542+0000 7f242afd6640 10 monclient: init 2024-03-25T19:36:31.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.543+0000 7f242afd6640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:31.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.543+0000 7f242afd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:31.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.543+0000 7f242afd6640 10 monclient(hunting): picked mon.noname-c con 0x7f24240b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:31.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.543+0000 7f242afd6640 10 monclient(hunting): picked mon.noname-b con 0x7f24240b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:31.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.543+0000 7f242afd6640 10 monclient(hunting): picked mon.noname-a con 0x7f242414be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:31.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.544+0000 7f242afd6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2424138870 con 0x7f242414be90 2024-03-25T19:36:31.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.544+0000 7f242afd6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24241d7810 con 0x7f24240b4d60 2024-03-25T19:36:31.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.544+0000 7f242afd6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24241d8440 con 0x7f24240b1730 2024-03-25T19:36:31.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.544+0000 7f242afd6640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:31.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.544+0000 7f242afd6640 10 monclient(hunting): authenticate will time out at 4583.884906s 2024-03-25T19:36:31.550 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.551+0000 7f2423fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f24240b4d60 0x7f2424136d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:40524/0 (socket says 172.21.15.107:40524) 2024-03-25T19:36:31.550 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.551+0000 7f2423fff640 1 -- 172.21.15.107:0/4050039012 learned_addr learned my addr 172.21.15.107:0/4050039012 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:31.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.552+0000 7f24217fa640 10 client.?.objecter ms_handle_connect 0x7f24240b4d60 2024-03-25T19:36:31.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.552+0000 7f24217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:31.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.559+0000 7f24217fa640 10 client.?.objecter ms_handle_connect 0x7f24240b1730 2024-03-25T19:36:31.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.559+0000 7f24217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:31.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.560+0000 7f24217fa640 10 client.?.objecter ms_handle_connect 0x7f242414be90 2024-03-25T19:36:31.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.560+0000 7f24217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:31.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.560+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368532479 0 0) 0x7f24241d7810 con 0x7f24240b4d60 2024-03-25T19:36:31.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.560+0000 7f24217fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:31.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.560+0000 7f24217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:31.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.560+0000 7f24217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:31.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.560+0000 7f24217fa640 10 monclient(hunting): my global_id is 4243 2024-03-25T19:36:31.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.560+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2408004eb0 con 0x7f24240b4d60 2024-03-25T19:36:31.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551793663 0 0) 0x7f24241d8440 con 0x7f24240b1730 2024-03-25T19:36:31.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:31.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:31.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:31.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:36:31.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24241d7810 con 0x7f24240b1730 2024-03-25T19:36:31.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235107072 0 0) 0x7f2424138870 con 0x7f242414be90 2024-03-25T19:36:31.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:31.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:31.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:31.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 10 monclient(hunting): my global_id is 4251 2024-03-25T19:36:31.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.570+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24241d8440 con 0x7f242414be90 2024-03-25T19:36:31.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.574+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1667404828 0 0) 0x7f2408004eb0 con 0x7f24240b4d60 2024-03-25T19:36:31.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.574+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2424138870 con 0x7f24240b4d60 2024-03-25T19:36:31.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.574+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2410001c20 con 0x7f24240b4d60 2024-03-25T19:36:31.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.574+0000 7f24217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:31.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.574+0000 7f24217fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:31.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.574+0000 7f24217fa640 10 monclient(hunting): dump: 2024-03-25T19:36:31.574 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:31.574 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:31.574 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:31.574 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:31.575 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:31.575 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:31.575 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:31.575 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:31.575 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:31.575 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:31.575 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.574+0000 7f24217fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:31.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.581+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 886966248 0 0) 0x7f24241d7810 con 0x7f24240b1730 2024-03-25T19:36:31.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.581+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2408004eb0 con 0x7f24240b1730 2024-03-25T19:36:31.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.581+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2418001950 con 0x7f24240b1730 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.581+0000 7f24217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.581+0000 7f24217fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.581+0000 7f24217fa640 10 monclient(hunting): dump: 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:31.581 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:31.582 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:31.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.588+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3474034465 0 0) 0x7f2424138870 con 0x7f24240b4d60 2024-03-25T19:36:31.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 >> v1:172.21.15.107:6790/0 conn(0x7f24240b1730 legacy=0x7f24241d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:31.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 >> v1:172.21.15.107:6789/0 conn(0x7f242414be90 legacy=0x7f242413b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:31.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f24217fa640 10 monclient: _finish_hunting 0 2024-03-25T19:36:31.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f24217fa640 1 monclient: found mon.b 2024-03-25T19:36:31.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f24217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:31.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f24217fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:31.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24241d85d0 con 0x7f24240b4d60 2024-03-25T19:36:31.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f24217fa640 10 monclient: _finish_auth 0 2024-03-25T19:36:31.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f24217fa640 10 monclient: _check_auth_tickets 2024-03-25T19:36:31.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f24217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:31.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f242afd6640 5 monclient: authenticate success, global_id 4243 2024-03-25T19:36:31.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f242afd6640 10 monclient: _renew_subs 2024-03-25T19:36:31.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f242afd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:31.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f242afd6640 1 -- 172.21.15.107:0/4050039012 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2424137430 con 0x7f24240b4d60 2024-03-25T19:36:31.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f242afd6640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:31.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f242afd6640 10 monclient: _renew_subs 2024-03-25T19:36:31.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f242afd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:31.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f242afd6640 1 -- 172.21.15.107:0/4050039012 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2424136360 con 0x7f24240b4d60 2024-03-25T19:36:31.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.589+0000 7f242afd6640 1 librados: init done 2024-03-25T19:36:31.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.590+0000 7f242afd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:31.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.590+0000 7f242afd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:31.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.590+0000 7f242afd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:31.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.590+0000 7f242afd6640 1 -- 172.21.15.107:0/4050039012 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24240627b0 con 0x7f24240b4d60 2024-03-25T19:36:31.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.882+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2410004a90 con 0x7f24240b4d60 2024-03-25T19:36:31.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.882+0000 7f24217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:31.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.882+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2410004de0 con 0x7f24240b4d60 2024-03-25T19:36:31.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.882+0000 7f24217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:31.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.882+0000 7f24217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:31.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.882+0000 7f24217fa640 10 monclient: dump: 2024-03-25T19:36:31.882 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:31.882 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:31.882 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:31.882 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:31.882 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:31.882 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:31.883 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:31.883 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:31.883 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:31.883 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:31.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.882+0000 7f2422ffd640 4 set_mon_vals no callback set 2024-03-25T19:36:31.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.883+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f2410005000 con 0x7f24240b4d60 2024-03-25T19:36:31.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.883+0000 7f2428d72640 10 monclient: get_auth_request con 0x7f240806dec0 auth_method 0 2024-03-25T19:36:31.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.884+0000 7f24217fa640 10 client.4243.objecter ms_handle_connect 0x7f240806dec0 2024-03-25T19:36:31.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.888+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f2410054960 con 0x7f24240b4d60 2024-03-25T19:36:31.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.888+0000 7f24217fa640 10 client.4243.objecter ms_dispatch 0x7f2424000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:31.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.888+0000 7f24217fa640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:31.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.888+0000 7f24217fa640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:31.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.889+0000 7f24217fa640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-25T19:36:31.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.889+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2410050470 con 0x7f24240b4d60 2024-03-25T19:36:31.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.889+0000 7f24217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:31.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:31.889+0000 7f24217fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:32.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.122+0000 7f242afd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:32.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.122+0000 7f242afd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:32.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.122+0000 7f242afd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:32.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.122+0000 7f242afd6640 1 -- 172.21.15.107:0/4050039012 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2424069f40 con 0x7f24240b4d60 2024-03-25T19:36:32.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.174+0000 7f24217fa640 1 -- 172.21.15.107:0/4050039012 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f24100523a0 con 0x7f24240b4d60 2024-03-25T19:36:32.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.175+0000 7f24217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:32.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.175+0000 7f24217fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:32.175 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:36:32.175 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:32.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 10 librados: watch_flush enter 2024-03-25T19:36:32.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 10 librados: watch_flush exit 2024-03-25T19:36:32.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-25T19:36:32.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 10 client.4243.objecter successfully canceled tick 2024-03-25T19:36:32.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 1 -- 172.21.15.107:0/4050039012 >> v1:172.21.15.195:6800/2984408421 conn(0x7f240806dec0 legacy=0x7f2408070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:32.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 10 monclient: shutdown 2024-03-25T19:36:32.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:32.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 1 -- 172.21.15.107:0/4050039012 >> v1:172.21.15.195:6789/0 conn(0x7f24240b4d60 legacy=0x7f2424136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:32.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 1 -- 172.21.15.107:0/4050039012 shutdown_connections 2024-03-25T19:36:32.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 1 -- 172.21.15.107:0/4050039012 >> 172.21.15.107:0/4050039012 conn(0x7f24240617a0 msgr2=0x7f242413c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:32.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 1 -- 172.21.15.107:0/4050039012 shutdown_connections 2024-03-25T19:36:32.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 1 -- 172.21.15.107:0/4050039012 wait complete. 2024-03-25T19:36:32.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:32.177+0000 7f242afd6640 1 librados: shutdown 2024-03-25T19:36:32.190 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:38.995 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:39.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.109+0000 7fd512220640 10 monclient: get_monmap_and_config 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.109+0000 7fd512220640 10 monclient: build_initial_monmap 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.109+0000 7fd512220640 10 monclient: monmap: 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:36:39.110473+0000 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:36:39.110473+0000 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:36:39.109 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:39.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.110+0000 7fd512220640 1 Processor -- start 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.110+0000 7fd512220640 1 -- start start 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.110+0000 7fd512220640 10 monclient: init 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.111+0000 7fd512220640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.111+0000 7fd512220640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.111+0000 7fd512220640 10 monclient(hunting): picked mon.noname-b con 0x7fd50c13fc60 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.111+0000 7fd512220640 10 monclient(hunting): picked mon.noname-c con 0x7fd50c070570 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.111+0000 7fd512220640 10 monclient(hunting): picked mon.noname-a con 0x7fd50c0727a0 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.111+0000 7fd512220640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c13c9f0 con 0x7fd50c0727a0 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.111+0000 7fd512220640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c06fac0 con 0x7fd50c070570 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.111+0000 7fd512220640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c0715d0 con 0x7fd50c13fc60 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.111+0000 7fd512220640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:39.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.111+0000 7fd512220640 10 monclient(hunting): authenticate will time out at 4591.451952s 2024-03-25T19:36:39.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.114+0000 7fd50affd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd50c070570 0x7fd50c072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:45004/0 (socket says 172.21.15.107:45004) 2024-03-25T19:36:39.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.114+0000 7fd50affd640 1 -- 172.21.15.107:0/2545228647 learned_addr learned my addr 172.21.15.107:0/2545228647 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:39.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.123+0000 7fd50a7fc640 1 -- 172.21.15.107:0/2545228647 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785233577 0 0) 0x7fd50c06fac0 con 0x7fd50c070570 2024-03-25T19:36:39.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.123+0000 7fd50a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:39.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.123+0000 7fd50a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:39.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.123+0000 7fd50a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:39.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.123+0000 7fd50a7fc640 10 monclient(hunting): my global_id is 4246 2024-03-25T19:36:39.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.123+0000 7fd50a7fc640 1 -- 172.21.15.107:0/2545228647 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f4005730 con 0x7fd50c070570 2024-03-25T19:36:39.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.124+0000 7fd50a7fc640 1 -- 172.21.15.107:0/2545228647 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991718404 0 0) 0x7fd50c13c9f0 con 0x7fd50c0727a0 2024-03-25T19:36:39.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.124+0000 7fd50a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:39.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.124+0000 7fd50a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:39.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.124+0000 7fd50a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:39.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.124+0000 7fd50a7fc640 10 monclient(hunting): my global_id is 4254 2024-03-25T19:36:39.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.124+0000 7fd50a7fc640 1 -- 172.21.15.107:0/2545228647 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50c06fac0 con 0x7fd50c0727a0 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 1 -- 172.21.15.107:0/2545228647 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3887293053 0 0) 0x7fd4f4005730 con 0x7fd50c070570 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 1 -- 172.21.15.107:0/2545228647 >> v1:172.21.15.107:6790/0 conn(0x7fd50c13fc60 legacy=0x7fd50c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 1 -- 172.21.15.107:0/2545228647 >> v1:172.21.15.107:6789/0 conn(0x7fd50c0727a0 legacy=0x7fd50c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 1 -- 172.21.15.107:0/2545228647 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c0718b0 con 0x7fd50c070570 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 1 -- 172.21.15.107:0/2545228647 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd500001690 con 0x7fd50c070570 2024-03-25T19:36:39.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd50a7fc640 10 monclient: dump: 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd512220640 5 monclient: authenticate success, global_id 4246 2024-03-25T19:36:39.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.128+0000 7fd512220640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:39.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.158+0000 7fd50a7fc640 1 -- 172.21.15.107:0/2545228647 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5000017f0 con 0x7fd50c070570 2024-03-25T19:36:39.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.158+0000 7fd50a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.158+0000 7fd50a7fc640 1 -- 172.21.15.107:0/2545228647 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd500001c00 con 0x7fd50c070570 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.158+0000 7fd512220640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.158+0000 7fd512220640 4 set_mon_vals no callback set 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.158+0000 7fd50a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.158+0000 7fd50a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.158+0000 7fd50a7fc640 10 monclient: dump: 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:39.158 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:39.159 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:39.159 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:39.159 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:39.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.159+0000 7fd512220640 10 monclient: shutdown 2024-03-25T19:36:39.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.159+0000 7fd512220640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:39.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.159+0000 7fd512220640 1 -- 172.21.15.107:0/2545228647 >> v1:172.21.15.195:6789/0 conn(0x7fd50c070570 legacy=0x7fd50c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.159+0000 7fd512220640 1 -- 172.21.15.107:0/2545228647 shutdown_connections 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.159+0000 7fd512220640 1 -- 172.21.15.107:0/2545228647 >> 172.21.15.107:0/2545228647 conn(0x7fd50c0b4a80 msgr2=0x7fd50c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.159+0000 7fd512220640 1 -- 172.21.15.107:0/2545228647 shutdown_connections 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.159+0000 7fd512220640 1 -- 172.21.15.107:0/2545228647 wait complete. 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.160+0000 7fd512220640 10 monclient: build_initial_monmap 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.160+0000 7fd512220640 10 monclient: monmap: 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:39.161 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:36:39.162 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:36:39.162 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:39.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.160+0000 7fd512220640 1 librados: starting msgr at 2024-03-25T19:36:39.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.160+0000 7fd512220640 1 librados: starting objecter 2024-03-25T19:36:39.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.160+0000 7fd512220640 1 Processor -- start 2024-03-25T19:36:39.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.161+0000 7fd512220640 1 -- start start 2024-03-25T19:36:39.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.161+0000 7fd512220640 1 librados: setting wanted keys 2024-03-25T19:36:39.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.161+0000 7fd512220640 1 librados: calling monclient init 2024-03-25T19:36:39.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.161+0000 7fd512220640 10 monclient: init 2024-03-25T19:36:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.161+0000 7fd512220640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.161+0000 7fd512220640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.161+0000 7fd512220640 10 monclient(hunting): picked mon.noname-a con 0x7fd50c070570 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.161+0000 7fd512220640 10 monclient(hunting): picked mon.noname-b con 0x7fd50c0727a0 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.161+0000 7fd512220640 10 monclient(hunting): picked mon.noname-c con 0x7fd50c13fc60 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.161+0000 7fd512220640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c142bf0 con 0x7fd50c070570 2024-03-25T19:36:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.161+0000 7fd512220640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c1d7e60 con 0x7fd50c0727a0 2024-03-25T19:36:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.162+0000 7fd512220640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c1428a0 con 0x7fd50c13fc60 2024-03-25T19:36:39.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.162+0000 7fd512220640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:39.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.162+0000 7fd512220640 10 monclient(hunting): authenticate will time out at 4591.502767s 2024-03-25T19:36:39.176 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.176+0000 7fd50affd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd50c0727a0 0x7fd50c133f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:45042/0 (socket says 172.21.15.107:45042) 2024-03-25T19:36:39.176 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.176+0000 7fd50affd640 1 -- 172.21.15.107:0/1212909398 learned_addr learned my addr 172.21.15.107:0/1212909398 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:39.176 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.177+0000 7fd4f3fff640 10 client.?.objecter ms_handle_connect 0x7fd50c0727a0 2024-03-25T19:36:39.176 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.177+0000 7fd4f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:39.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.182+0000 7fd4f3fff640 10 client.?.objecter ms_handle_connect 0x7fd50c070570 2024-03-25T19:36:39.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.182+0000 7fd4f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:39.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.182+0000 7fd4f3fff640 10 client.?.objecter ms_handle_connect 0x7fd50c13fc60 2024-03-25T19:36:39.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.182+0000 7fd4f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:39.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.190+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342795368 0 0) 0x7fd50c1d7e60 con 0x7fd50c0727a0 2024-03-25T19:36:39.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.190+0000 7fd4f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:39.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.190+0000 7fd4f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:39.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.190+0000 7fd4f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:39.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.190+0000 7fd4f3fff640 10 monclient(hunting): my global_id is 4249 2024-03-25T19:36:39.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.190+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f8004eb0 con 0x7fd50c0727a0 2024-03-25T19:36:39.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577289899 0 0) 0x7fd50c142bf0 con 0x7fd50c070570 2024-03-25T19:36:39.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:39.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:39.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:39.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:36:39.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50c1d7e60 con 0x7fd50c070570 2024-03-25T19:36:39.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950302092 0 0) 0x7fd50c1428a0 con 0x7fd50c13fc60 2024-03-25T19:36:39.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:39.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:39.191 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:39.192 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:36:39.192 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.191+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50c142bf0 con 0x7fd50c13fc60 2024-03-25T19:36:39.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3621305 0 0) 0x7fd50c142bf0 con 0x7fd50c13fc60 2024-03-25T19:36:39.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd50c1428a0 con 0x7fd50c13fc60 2024-03-25T19:36:39.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd504001cb0 con 0x7fd50c13fc60 2024-03-25T19:36:39.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:39.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:39.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 10 monclient(hunting): dump: 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2600910938 0 0) 0x7fd50c1d7e60 con 0x7fd50c070570 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd50c142bf0 con 0x7fd50c070570 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd4fc001770 con 0x7fd50c070570 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:39.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:39.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.201+0000 7fd4f3fff640 10 monclient(hunting): dump: 2024-03-25T19:36:39.203 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:39.203 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:39.203 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.203 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.203 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:39.203 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:39.203 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:39.203 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:39.203 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:39.203 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:39.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.207+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1403053014 0 0) 0x7fd4f8004eb0 con 0x7fd50c0727a0 2024-03-25T19:36:39.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.207+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd50c1d7e60 con 0x7fd50c0727a0 2024-03-25T19:36:39.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.207+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd500001c00 con 0x7fd50c0727a0 2024-03-25T19:36:39.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.207+0000 7fd4f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:39.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.207+0000 7fd4f3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:39.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.207+0000 7fd4f3fff640 10 monclient(hunting): dump: 2024-03-25T19:36:39.207 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:39.207 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:39.208 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.208 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.208 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:39.208 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:39.208 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:39.208 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:39.208 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:39.208 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:39.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.215+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3450078254 0 0) 0x7fd50c1428a0 con 0x7fd50c13fc60 2024-03-25T19:36:39.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 >> v1:172.21.15.195:6789/0 conn(0x7fd50c0727a0 legacy=0x7fd50c133f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:39.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 >> v1:172.21.15.107:6789/0 conn(0x7fd50c070570 legacy=0x7fd50c1d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:39.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd4f3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:39.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd4f3fff640 1 monclient: found mon.c 2024-03-25T19:36:39.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd4f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:39.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd4f3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:36:39.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c143110 con 0x7fd50c13fc60 2024-03-25T19:36:39.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd4f3fff640 10 monclient: _finish_auth 0 2024-03-25T19:36:39.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd4f3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:39.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd4f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:39.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd512220640 5 monclient: authenticate success, global_id 4259 2024-03-25T19:36:39.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd512220640 10 monclient: _renew_subs 2024-03-25T19:36:39.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd512220640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:36:39.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd512220640 1 -- 172.21.15.107:0/1212909398 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50c1433e0 con 0x7fd50c13fc60 2024-03-25T19:36:39.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd512220640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:39.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd512220640 10 monclient: _renew_subs 2024-03-25T19:36:39.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd512220640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:36:39.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd512220640 1 -- 172.21.15.107:0/1212909398 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50c13ec70 con 0x7fd50c13fc60 2024-03-25T19:36:39.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.216+0000 7fd512220640 1 librados: init done 2024-03-25T19:36:39.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.217+0000 7fd512220640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:39.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.217+0000 7fd512220640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:39.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.217+0000 7fd512220640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:36:39.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.217+0000 7fd512220640 1 -- 172.21.15.107:0/1212909398 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50c142a30 con 0x7fd50c13fc60 2024-03-25T19:36:39.535 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.535+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd504001310 con 0x7fd50c13fc60 2024-03-25T19:36:39.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.535+0000 7fd4f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:39.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.535+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd504003b10 con 0x7fd50c13fc60 2024-03-25T19:36:39.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.535+0000 7fd4f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:39.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.535+0000 7fd4f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:39.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.535+0000 7fd4f3fff640 10 monclient: dump: 2024-03-25T19:36:39.536 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:39.536 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:39.536 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.536 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:39.536 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:39.537 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:39.537 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:39.537 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:39.537 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:39.537 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:39.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.535+0000 7fd509ffb640 4 set_mon_vals no callback set 2024-03-25T19:36:39.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.536+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fd504019ae0 con 0x7fd50c13fc60 2024-03-25T19:36:39.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.537+0000 7fd50b7fe640 10 monclient: get_auth_request con 0x7fd4f806e0b0 auth_method 0 2024-03-25T19:36:39.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.537+0000 7fd4f3fff640 10 client.4259.objecter ms_handle_connect 0x7fd4f806e0b0 2024-03-25T19:36:39.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.543+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fd504053830 con 0x7fd50c13fc60 2024-03-25T19:36:39.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.543+0000 7fd4f3fff640 10 client.4259.objecter ms_dispatch 0x7fd50c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:39.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.543+0000 7fd4f3fff640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:39.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.543+0000 7fd4f3fff640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:39.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.543+0000 7fd4f3fff640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-25T19:36:39.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.543+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd50404f170 con 0x7fd50c13fc60 2024-03-25T19:36:39.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.543+0000 7fd4f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:39.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.543+0000 7fd4f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:39.789 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.789+0000 7fd512220640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:39.789 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.789+0000 7fd512220640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:39.789 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.789+0000 7fd512220640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:36:39.789 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.789+0000 7fd512220640 1 -- 172.21.15.107:0/1212909398 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd50c142d80 con 0x7fd50c13fc60 2024-03-25T19:36:39.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.822+0000 7fd4f3fff640 1 -- 172.21.15.107:0/1212909398 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fd504051270 con 0x7fd50c13fc60 2024-03-25T19:36:39.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.822+0000 7fd4f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:39.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.822+0000 7fd4f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:39.822 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:36:39.822 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:39.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 10 librados: watch_flush enter 2024-03-25T19:36:39.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 10 librados: watch_flush exit 2024-03-25T19:36:39.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-25T19:36:39.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 10 client.4259.objecter successfully canceled tick 2024-03-25T19:36:39.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 1 -- 172.21.15.107:0/1212909398 >> v1:172.21.15.195:6800/2984408421 conn(0x7fd4f806e0b0 legacy=0x7fd4f8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:39.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 10 monclient: shutdown 2024-03-25T19:36:39.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:39.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 1 -- 172.21.15.107:0/1212909398 >> v1:172.21.15.107:6790/0 conn(0x7fd50c13fc60 legacy=0x7fd50c1e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:39.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 1 -- 172.21.15.107:0/1212909398 shutdown_connections 2024-03-25T19:36:39.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 1 -- 172.21.15.107:0/1212909398 >> 172.21.15.107:0/1212909398 conn(0x7fd50c0b4a80 msgr2=0x7fd50c0ae910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:39.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 1 -- 172.21.15.107:0/1212909398 shutdown_connections 2024-03-25T19:36:39.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 1 -- 172.21.15.107:0/1212909398 wait complete. 2024-03-25T19:36:39.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:39.824+0000 7fd512220640 1 librados: shutdown 2024-03-25T19:36:39.835 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:44.346 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:01:14.662 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.411 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:01:14.728 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.450 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:01:14.737 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.510 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:01:14.797 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.564 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:01:14.880 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.626 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:01:14.942 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.652 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:01:14.968 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.721 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:01:15.008 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:44.861 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:01:15.178 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:46.641 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:46.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.757+0000 7f9438e7b640 10 monclient: get_monmap_and_config 2024-03-25T19:36:46.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.757+0000 7f9438e7b640 10 monclient: build_initial_monmap 2024-03-25T19:36:46.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.757+0000 7f9438e7b640 10 monclient: monmap: 2024-03-25T19:36:46.757 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:46.757 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:46.758 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:36:46.758712+0000 2024-03-25T19:36:46.758 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:36:46.758712+0000 2024-03-25T19:36:46.758 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:46.758 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:46.758 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:46.758 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:36:46.758 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:36:46.758 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:46.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.758+0000 7f9438e7b640 1 Processor -- start 2024-03-25T19:36:46.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.758+0000 7f9438e7b640 1 -- start start 2024-03-25T19:36:46.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.758+0000 7f9438e7b640 10 monclient: init 2024-03-25T19:36:46.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.759+0000 7f9438e7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:46.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.759+0000 7f9438e7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:46.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.759+0000 7f9438e7b640 10 monclient(hunting): picked mon.noname-c con 0x7f94340a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:46.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.759+0000 7f9438e7b640 10 monclient(hunting): picked mon.noname-b con 0x7f943413f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:46.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.759+0000 7f9438e7b640 10 monclient(hunting): picked mon.noname-a con 0x7f94341430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:46.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.759+0000 7f9438e7b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9434142520 con 0x7f94341430c0 2024-03-25T19:36:46.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.759+0000 7f9438e7b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94340b28a0 con 0x7f94340a98f0 2024-03-25T19:36:46.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.759+0000 7f9438e7b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94340b2a30 con 0x7f943413f710 2024-03-25T19:36:46.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.759+0000 7f9438e7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:46.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.759+0000 7f9438e7b640 10 monclient(hunting): authenticate will time out at 4599.100251s 2024-03-25T19:36:46.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.774+0000 7f943259c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f94340a98f0 0x7f94340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:42470/0 (socket says 172.21.15.107:42470) 2024-03-25T19:36:46.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.774+0000 7f943259c640 1 -- 172.21.15.107:0/132151129 learned_addr learned my addr 172.21.15.107:0/132151129 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:46.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.791+0000 7f9431d9b640 1 -- 172.21.15.107:0/132151129 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839341649 0 0) 0x7f94340b28a0 con 0x7f94340a98f0 2024-03-25T19:36:46.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.791+0000 7f9431d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:46.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.791+0000 7f9431d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:46.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.791+0000 7f9431d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:46.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.791+0000 7f9431d9b640 10 monclient(hunting): my global_id is 4252 2024-03-25T19:36:46.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.791+0000 7f9431d9b640 1 -- 172.21.15.107:0/132151129 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9420003520 con 0x7f94340a98f0 2024-03-25T19:36:46.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.794+0000 7f9431d9b640 1 -- 172.21.15.107:0/132151129 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840445560 0 0) 0x7f94340b2a30 con 0x7f943413f710 2024-03-25T19:36:46.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.794+0000 7f9431d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:46.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.794+0000 7f9431d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:46.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.794+0000 7f9431d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:46.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.794+0000 7f9431d9b640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:36:46.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.794+0000 7f9431d9b640 1 -- 172.21.15.107:0/132151129 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94340b28a0 con 0x7f943413f710 2024-03-25T19:36:46.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 1 -- 172.21.15.107:0/132151129 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 156840540 0 0) 0x7f9420003520 con 0x7f94340a98f0 2024-03-25T19:36:46.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 1 -- 172.21.15.107:0/132151129 >> v1:172.21.15.107:6790/0 conn(0x7f943413f710 legacy=0x7f94341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:46.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 1 -- 172.21.15.107:0/132151129 >> v1:172.21.15.107:6789/0 conn(0x7f94341430c0 legacy=0x7f943414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:46.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:36:46.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 1 monclient: found mon.noname-c 2024-03-25T19:36:46.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:46.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:36:46.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 1 -- 172.21.15.107:0/132151129 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94340b44c0 con 0x7f94340a98f0 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 10 monclient: _finish_auth 0 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 1 -- 172.21.15.107:0/132151129 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f9424001690 con 0x7f94340a98f0 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9438e7b640 5 monclient: authenticate success, global_id 4252 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9438e7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9431d9b640 10 monclient: dump: 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:46.805 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:46.806 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:46.806 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:46.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.804+0000 7f9438e7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:46.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.827+0000 7f9431d9b640 1 -- 172.21.15.107:0/132151129 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94240018b0 con 0x7f94340a98f0 2024-03-25T19:36:46.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.827+0000 7f9431d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:46.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.827+0000 7f9431d9b640 1 -- 172.21.15.107:0/132151129 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f9424001c40 con 0x7f94340a98f0 2024-03-25T19:36:46.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.827+0000 7f9438e7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.827+0000 7f9438e7b640 4 set_mon_vals no callback set 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.827+0000 7f9431d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.827+0000 7f9431d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.827+0000 7f9431d9b640 10 monclient: dump: 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:46.827 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:46.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.828+0000 7f9438e7b640 10 monclient: shutdown 2024-03-25T19:36:46.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.828+0000 7f9438e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.828+0000 7f9438e7b640 1 -- 172.21.15.107:0/132151129 >> v1:172.21.15.195:6789/0 conn(0x7f94340a98f0 legacy=0x7f94340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.828+0000 7f9438e7b640 1 -- 172.21.15.107:0/132151129 shutdown_connections 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.828+0000 7f9438e7b640 1 -- 172.21.15.107:0/132151129 >> 172.21.15.107:0/132151129 conn(0x7f94340b7ea0 msgr2=0x7f94340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.828+0000 7f9438e7b640 1 -- 172.21.15.107:0/132151129 shutdown_connections 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.828+0000 7f9438e7b640 1 -- 172.21.15.107:0/132151129 wait complete. 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.829+0000 7f9438e7b640 10 monclient: build_initial_monmap 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.829+0000 7f9438e7b640 10 monclient: monmap: 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:36:46.829 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:36:46.830 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:46.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.829+0000 7f9438e7b640 1 librados: starting msgr at 2024-03-25T19:36:46.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.829+0000 7f9438e7b640 1 librados: starting objecter 2024-03-25T19:36:46.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.830+0000 7f9438e7b640 1 Processor -- start 2024-03-25T19:36:46.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.830+0000 7f9438e7b640 1 -- start start 2024-03-25T19:36:46.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.830+0000 7f9438e7b640 1 librados: setting wanted keys 2024-03-25T19:36:46.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.830+0000 7f9438e7b640 1 librados: calling monclient init 2024-03-25T19:36:46.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.830+0000 7f9438e7b640 10 monclient: init 2024-03-25T19:36:46.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.830+0000 7f9438e7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:46.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.831+0000 7f9438e7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:46.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.831+0000 7f9438e7b640 10 monclient(hunting): picked mon.noname-a con 0x7f94340a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:46.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.831+0000 7f9438e7b640 10 monclient(hunting): picked mon.noname-b con 0x7f943413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:46.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.831+0000 7f9438e7b640 10 monclient(hunting): picked mon.noname-c con 0x7f94341430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:46.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.831+0000 7f9438e7b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9434139a40 con 0x7f94340a98f0 2024-03-25T19:36:46.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.831+0000 7f9438e7b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9434133f20 con 0x7f943413f710 2024-03-25T19:36:46.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.831+0000 7f9438e7b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94340ab950 con 0x7f94341430c0 2024-03-25T19:36:46.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.831+0000 7f9438e7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:46.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.831+0000 7f9438e7b640 10 monclient(hunting): authenticate will time out at 4599.172034s 2024-03-25T19:36:46.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.844+0000 7f9432d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f94341430c0 0x7f9434133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54254/0 (socket says 172.21.15.107:54254) 2024-03-25T19:36:46.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.844+0000 7f9432d9d640 1 -- 172.21.15.107:0/3545231533 learned_addr learned my addr 172.21.15.107:0/3545231533 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:46.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.845+0000 7f942affd640 10 client.?.objecter ms_handle_connect 0x7f943413f710 2024-03-25T19:36:46.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.846+0000 7f942affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:46.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.847+0000 7f942affd640 10 client.?.objecter ms_handle_connect 0x7f94341430c0 2024-03-25T19:36:46.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.847+0000 7f942affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:46.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.854+0000 7f942affd640 10 client.?.objecter ms_handle_connect 0x7f94340a98f0 2024-03-25T19:36:46.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.854+0000 7f942affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:46.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.856+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845666634 0 0) 0x7f9434133f20 con 0x7f943413f710 2024-03-25T19:36:46.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.856+0000 7f942affd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:46.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.856+0000 7f942affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:46.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.856+0000 7f942affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:46.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.856+0000 7f942affd640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:36:46.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.856+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9414004eb0 con 0x7f943413f710 2024-03-25T19:36:46.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.861+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712352733 0 0) 0x7f9434139a40 con 0x7f94340a98f0 2024-03-25T19:36:46.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.861+0000 7f942affd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:46.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.861+0000 7f942affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:46.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.861+0000 7f942affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:46.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.861+0000 7f942affd640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:36:46.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.861+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9434133f20 con 0x7f94340a98f0 2024-03-25T19:36:46.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.864+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048630392 0 0) 0x7f94340ab950 con 0x7f94341430c0 2024-03-25T19:36:46.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.864+0000 7f942affd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:46.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.864+0000 7f942affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:46.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.864+0000 7f942affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:46.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.864+0000 7f942affd640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:36:46.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.864+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9434139a40 con 0x7f94341430c0 2024-03-25T19:36:46.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.873+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3491130058 0 0) 0x7f9414004eb0 con 0x7f943413f710 2024-03-25T19:36:46.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.873+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94340ab950 con 0x7f943413f710 2024-03-25T19:36:46.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.873+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f941c001930 con 0x7f943413f710 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.873+0000 7f942affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.873+0000 7f942affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.873+0000 7f942affd640 10 monclient(hunting): dump: 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:46.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.873+0000 7f942affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:46.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 519176879 0 0) 0x7f94340ab950 con 0x7f943413f710 2024-03-25T19:36:46.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 >> v1:172.21.15.107:6790/0 conn(0x7f94341430c0 legacy=0x7f9434133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:46.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 >> v1:172.21.15.107:6789/0 conn(0x7f94340a98f0 legacy=0x7f9434138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:46.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f942affd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:46.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f942affd640 1 monclient: found mon.b 2024-03-25T19:36:46.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f942affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:46.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f942affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:46.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94340abc30 con 0x7f943413f710 2024-03-25T19:36:46.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f942affd640 10 monclient: _finish_auth 0 2024-03-25T19:36:46.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f942affd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:46.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f942affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:46.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f9438e7b640 5 monclient: authenticate success, global_id 4255 2024-03-25T19:36:46.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f9438e7b640 10 monclient: _renew_subs 2024-03-25T19:36:46.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f9438e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:46.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f9438e7b640 1 -- 172.21.15.107:0/3545231533 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94340abf10 con 0x7f943413f710 2024-03-25T19:36:46.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f9438e7b640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:46.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f9438e7b640 10 monclient: _renew_subs 2024-03-25T19:36:46.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f9438e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:46.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f9438e7b640 1 -- 172.21.15.107:0/3545231533 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9434139370 con 0x7f943413f710 2024-03-25T19:36:46.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.882+0000 7f9438e7b640 1 librados: init done 2024-03-25T19:36:46.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.883+0000 7f9438e7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:46.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.883+0000 7f9438e7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:46.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.883+0000 7f9438e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:46.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:46.883+0000 7f9438e7b640 1 -- 172.21.15.107:0/3545231533 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94340627b0 con 0x7f943413f710 2024-03-25T19:36:47.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.162+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f941c003610 con 0x7f943413f710 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.162+0000 7f942affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.162+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f941c003a20 con 0x7f943413f710 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.162+0000 7f942affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.162+0000 7f942affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.162+0000 7f942affd640 10 monclient: dump: 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:47.162 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:47.163 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:47.163 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:47.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.162+0000 7f943159a640 4 set_mon_vals no callback set 2024-03-25T19:36:47.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.162+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f941c003c40 con 0x7f943413f710 2024-03-25T19:36:47.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.163+0000 7f943259c640 10 monclient: get_auth_request con 0x7f941406ddf0 auth_method 0 2024-03-25T19:36:47.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.163+0000 7f942affd640 10 client.4255.objecter ms_handle_connect 0x7f941406ddf0 2024-03-25T19:36:47.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.167+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f941c053660 con 0x7f943413f710 2024-03-25T19:36:47.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.167+0000 7f942affd640 10 client.4255.objecter ms_dispatch 0x7f9434000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:47.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.167+0000 7f942affd640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:47.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.167+0000 7f942affd640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:47.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.167+0000 7f942affd640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-25T19:36:47.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.167+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f941c04f170 con 0x7f943413f710 2024-03-25T19:36:47.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.167+0000 7f942affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:47.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.167+0000 7f942affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:47.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.414+0000 7f9438e7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:47.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.414+0000 7f9438e7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:47.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.414+0000 7f9438e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:47.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.414+0000 7f9438e7b640 1 -- 172.21.15.107:0/3545231533 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94340b7180 con 0x7f943413f710 2024-03-25T19:36:47.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.452+0000 7f942affd640 1 -- 172.21.15.107:0/3545231533 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f941c0510a0 con 0x7f943413f710 2024-03-25T19:36:47.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.452+0000 7f942affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:47.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.452+0000 7f942affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:47.452 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:36:47.452 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.454+0000 7f9438e7b640 10 librados: watch_flush enter 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.454+0000 7f9438e7b640 10 librados: watch_flush exit 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.454+0000 7f9438e7b640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.454+0000 7f9438e7b640 10 client.4255.objecter successfully canceled tick 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.454+0000 7f9438e7b640 1 -- 172.21.15.107:0/3545231533 >> v1:172.21.15.195:6800/2984408421 conn(0x7f941406ddf0 legacy=0x7f94140702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.454+0000 7f9438e7b640 10 monclient: shutdown 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.454+0000 7f9438e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.454+0000 7f9438e7b640 1 -- 172.21.15.107:0/3545231533 >> v1:172.21.15.195:6789/0 conn(0x7f943413f710 legacy=0x7f9434133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.454+0000 7f9438e7b640 1 -- 172.21.15.107:0/3545231533 shutdown_connections 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.454+0000 7f9438e7b640 1 -- 172.21.15.107:0/3545231533 >> 172.21.15.107:0/3545231533 conn(0x7f94340b7ea0 msgr2=0x7f9434074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.454+0000 7f9438e7b640 1 -- 172.21.15.107:0/3545231533 shutdown_connections 2024-03-25T19:36:47.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.455+0000 7f9438e7b640 1 -- 172.21.15.107:0/3545231533 wait complete. 2024-03-25T19:36:47.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:47.455+0000 7f9438e7b640 1 librados: shutdown 2024-03-25T19:36:47.464 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:54.271 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:54.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.366+0000 7f4307d6d640 10 monclient: get_monmap_and_config 2024-03-25T19:36:54.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.366+0000 7f4307d6d640 10 monclient: build_initial_monmap 2024-03-25T19:36:54.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.366+0000 7f4307d6d640 10 monclient: monmap: 2024-03-25T19:36:54.366 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:54.366 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:54.366 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:36:54.367475+0000 2024-03-25T19:36:54.366 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:36:54.367475+0000 2024-03-25T19:36:54.366 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:54.366 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:54.366 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:54.366 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:36:54.367 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:36:54.367 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.367+0000 7f4307d6d640 1 Processor -- start 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.367+0000 7f4307d6d640 1 -- start start 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.367+0000 7f4307d6d640 10 monclient: init 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.367+0000 7f4307d6d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.367+0000 7f4307d6d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.367+0000 7f4307d6d640 10 monclient(hunting): picked mon.noname-a con 0x7f43000a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.367+0000 7f4307d6d640 10 monclient(hunting): picked mon.noname-c con 0x7f430013f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.368+0000 7f4307d6d640 10 monclient(hunting): picked mon.noname-b con 0x7f43001430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.368+0000 7f4307d6d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4300142520 con 0x7f43000a98f0 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.368+0000 7f4307d6d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43000b28a0 con 0x7f430013f710 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.368+0000 7f4307d6d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43000b2a30 con 0x7f43001430c0 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.368+0000 7f4307d6d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:54.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.368+0000 7f4307d6d640 10 monclient(hunting): authenticate will time out at 4606.708837s 2024-03-25T19:36:54.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.379+0000 7f430630a640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f43001430c0 0x7f430014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42162/0 (socket says 172.21.15.107:42162) 2024-03-25T19:36:54.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.379+0000 7f430630a640 1 -- 172.21.15.107:0/2367793574 learned_addr learned my addr 172.21.15.107:0/2367793574 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:54.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.408+0000 7f4304b07640 1 -- 172.21.15.107:0/2367793574 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364201109 0 0) 0x7f43000b2a30 con 0x7f43001430c0 2024-03-25T19:36:54.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.408+0000 7f4304b07640 10 monclient(hunting): _negotiate 2024-03-25T19:36:54.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.408+0000 7f4304b07640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:54.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.408+0000 7f4304b07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:54.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.408+0000 7f4304b07640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:36:54.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.408+0000 7f4304b07640 1 -- 172.21.15.107:0/2367793574 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42ec003520 con 0x7f43001430c0 2024-03-25T19:36:54.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 1 -- 172.21.15.107:0/2367793574 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3258125752 0 0) 0x7f42ec003520 con 0x7f43001430c0 2024-03-25T19:36:54.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 1 -- 172.21.15.107:0/2367793574 >> v1:172.21.15.195:6789/0 conn(0x7f430013f710 legacy=0x7f43001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:36:54.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 1 -- 172.21.15.107:0/2367793574 >> v1:172.21.15.107:6789/0 conn(0x7f43000a98f0 legacy=0x7f43000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:54.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 10 monclient: _finish_hunting 0 2024-03-25T19:36:54.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 1 monclient: found mon.noname-b 2024-03-25T19:36:54.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:54.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:36:54.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 1 -- 172.21.15.107:0/2367793574 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43000b44c0 con 0x7f43001430c0 2024-03-25T19:36:54.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 10 monclient: _finish_auth 0 2024-03-25T19:36:54.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 10 monclient: _check_auth_tickets 2024-03-25T19:36:54.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 1 -- 172.21.15.107:0/2367793574 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f42f8001a10 con 0x7f43001430c0 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4307d6d640 5 monclient: authenticate success, global_id 4268 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4307d6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4304b07640 10 monclient: dump: 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:54.425 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:54.426 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:54.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.424+0000 7f4307d6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:54.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.444+0000 7f4304b07640 1 -- 172.21.15.107:0/2367793574 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42f8001be0 con 0x7f43001430c0 2024-03-25T19:36:54.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.444+0000 7f4304b07640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.444+0000 7f4304b07640 1 -- 172.21.15.107:0/2367793574 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f42f8002030 con 0x7f43001430c0 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.444+0000 7f4304b07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.444+0000 7f4304b07640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.444+0000 7f4304b07640 10 monclient: dump: 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:54.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.444+0000 7f4307d6d640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:54.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.444+0000 7f4307d6d640 4 set_mon_vals no callback set 2024-03-25T19:36:54.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.444+0000 7f4307d6d640 10 monclient: shutdown 2024-03-25T19:36:54.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.444+0000 7f4307d6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:54.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.445+0000 7f4307d6d640 1 -- 172.21.15.107:0/2367793574 >> v1:172.21.15.107:6790/0 conn(0x7f43001430c0 legacy=0x7f430014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:54.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.445+0000 7f4307d6d640 1 -- 172.21.15.107:0/2367793574 shutdown_connections 2024-03-25T19:36:54.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.445+0000 7f4307d6d640 1 -- 172.21.15.107:0/2367793574 >> 172.21.15.107:0/2367793574 conn(0x7f43000b7ea0 msgr2=0x7f43000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:54.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.445+0000 7f4307d6d640 1 -- 172.21.15.107:0/2367793574 shutdown_connections 2024-03-25T19:36:54.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.445+0000 7f4307d6d640 1 -- 172.21.15.107:0/2367793574 wait complete. 2024-03-25T19:36:54.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.445+0000 7f4307d6d640 10 monclient: build_initial_monmap 2024-03-25T19:36:54.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.445+0000 7f4307d6d640 10 monclient: monmap: 2024-03-25T19:36:54.450 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:36:54.450 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 1 librados: starting msgr at 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 1 librados: starting objecter 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 1 Processor -- start 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 1 -- start start 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 1 librados: setting wanted keys 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 1 librados: calling monclient init 2024-03-25T19:36:54.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 10 monclient: init 2024-03-25T19:36:54.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:54.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:54.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 10 monclient(hunting): picked mon.noname-b con 0x7f43000a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:36:54.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 10 monclient(hunting): picked mon.noname-a con 0x7f430013f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:36:54.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 10 monclient(hunting): picked mon.noname-c con 0x7f43001430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:36:54.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43001d77a0 con 0x7f430013f710 2024-03-25T19:36:54.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.446+0000 7f4307d6d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43001d8340 con 0x7f43000a98f0 2024-03-25T19:36:54.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.447+0000 7f4307d6d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43000abb90 con 0x7f43001430c0 2024-03-25T19:36:54.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.447+0000 7f4307d6d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:54.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.447+0000 7f4307d6d640 10 monclient(hunting): authenticate will time out at 4606.787769s 2024-03-25T19:36:54.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.453+0000 7f4305b09640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f43000a98f0 0x7f4300138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:52676/0 (socket says 172.21.15.107:52676) 2024-03-25T19:36:54.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.453+0000 7f4305b09640 1 -- 172.21.15.107:0/229315361 learned_addr learned my addr 172.21.15.107:0/229315361 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:36:54.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.454+0000 7f42f67fc640 10 client.?.objecter ms_handle_connect 0x7f43000a98f0 2024-03-25T19:36:54.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.454+0000 7f42f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:54.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.465+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144589385 0 0) 0x7f43001d8340 con 0x7f43000a98f0 2024-03-25T19:36:54.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.465+0000 7f42f67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:54.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.465+0000 7f42f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:54.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.465+0000 7f42f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:54.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.465+0000 7f42f67fc640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:36:54.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.465+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e4004eb0 con 0x7f43000a98f0 2024-03-25T19:36:54.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.468+0000 7f42f67fc640 10 client.?.objecter ms_handle_connect 0x7f43001430c0 2024-03-25T19:36:54.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.468+0000 7f42f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:54.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.469+0000 7f42f67fc640 10 client.?.objecter ms_handle_connect 0x7f430013f710 2024-03-25T19:36:54.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.469+0000 7f42f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:54.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.473+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1209565126 0 0) 0x7f42e4004eb0 con 0x7f43000a98f0 2024-03-25T19:36:54.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.473+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43001d8340 con 0x7f43000a98f0 2024-03-25T19:36:54.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.476+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732696375 0 0) 0x7f43000abb90 con 0x7f43001430c0 2024-03-25T19:36:54.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.476+0000 7f42f67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:54.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.476+0000 7f42f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:54.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.476+0000 7f42f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:54.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.476+0000 7f42f67fc640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:36:54.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.476+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e4004eb0 con 0x7f43001430c0 2024-03-25T19:36:54.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.482+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716489581 0 0) 0x7f43001d77a0 con 0x7f430013f710 2024-03-25T19:36:54.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.482+0000 7f42f67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:54.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.482+0000 7f42f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:54.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.482+0000 7f42f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:54.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.482+0000 7f42f67fc640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:36:54.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.483+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43000abb90 con 0x7f430013f710 2024-03-25T19:36:54.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.486+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4023947788 0 0) 0x7f42e4004eb0 con 0x7f43001430c0 2024-03-25T19:36:54.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.486+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43001d77a0 con 0x7f43001430c0 2024-03-25T19:36:54.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.486+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f42f8001d50 con 0x7f43001430c0 2024-03-25T19:36:54.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.486+0000 7f42f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:54.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.486+0000 7f42f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:54.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.486+0000 7f42f67fc640 10 monclient(hunting): dump: 2024-03-25T19:36:54.487 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:54.487 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:54.487 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:54.487 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:54.487 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:54.487 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:54.487 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:54.487 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:54.487 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:54.487 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:54.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.486+0000 7f42f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:54.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f42fc001b10 con 0x7f43000a98f0 2024-03-25T19:36:54.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:54.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:54.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 10 monclient(hunting): dump: 2024-03-25T19:36:54.495 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:54.495 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:54.495 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:54.495 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4167110273 0 0) 0x7f43001d8340 con 0x7f43000a98f0 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 >> v1:172.21.15.107:6790/0 conn(0x7f43001430c0 legacy=0x7f43001e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 >> v1:172.21.15.107:6789/0 conn(0x7f430013f710 legacy=0x7f43001d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 1 monclient: found mon.b 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:54.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4300137870 con 0x7f43000a98f0 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f42f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f4307d6d640 5 monclient: authenticate success, global_id 4258 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f4307d6d640 10 monclient: _renew_subs 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f4307d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f4307d6d640 1 -- 172.21.15.107:0/229315361 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4300137b20 con 0x7f43000a98f0 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f4307d6d640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f4307d6d640 10 monclient: _renew_subs 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f4307d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f4307d6d640 1 -- 172.21.15.107:0/229315361 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4300135220 con 0x7f43000a98f0 2024-03-25T19:36:54.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.495+0000 7f4307d6d640 1 librados: init done 2024-03-25T19:36:54.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.496+0000 7f4307d6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:54.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.496+0000 7f4307d6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:54.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.496+0000 7f4307d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:54.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.496+0000 7f4307d6d640 1 -- 172.21.15.107:0/229315361 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43000ab9b0 con 0x7f43000a98f0 2024-03-25T19:36:54.781 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.781+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42fc001c70 con 0x7f43000a98f0 2024-03-25T19:36:54.781 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.781+0000 7f42f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:54.781 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.781+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f42fc003a60 con 0x7f43000a98f0 2024-03-25T19:36:54.781 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.781+0000 7f42f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:54.781 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.781+0000 7f42f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:54.781 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.781+0000 7f42f67fc640 10 monclient: dump: 2024-03-25T19:36:54.781 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:36:54.781 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:36:54.781 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:54.781 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:36:54.781 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:54.782 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:36:54.782 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:36:54.782 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:36:54.782 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:36:54.782 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:36:54.782 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.781+0000 7f42f7fff640 4 set_mon_vals no callback set 2024-03-25T19:36:54.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.782+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f42fc003c80 con 0x7f43000a98f0 2024-03-25T19:36:54.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.783+0000 7f4305308640 10 monclient: get_auth_request con 0x7f42e406dfb0 auth_method 0 2024-03-25T19:36:54.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.783+0000 7f42f67fc640 10 client.4258.objecter ms_handle_connect 0x7f42e406dfb0 2024-03-25T19:36:54.786 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.786+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f42fc0548c0 con 0x7f43000a98f0 2024-03-25T19:36:54.786 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.786+0000 7f42f67fc640 10 client.4258.objecter ms_dispatch 0x7f4300000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:54.786 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.786+0000 7f42f67fc640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:54.786 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.786+0000 7f42f67fc640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:54.786 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.786+0000 7f42f67fc640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-25T19:36:54.786 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.787+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42fc0503d0 con 0x7f43000a98f0 2024-03-25T19:36:54.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.787+0000 7f42f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:54.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:54.787+0000 7f42f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:55.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.037+0000 7f4307d6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:55.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.037+0000 7f4307d6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:55.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.037+0000 7f4307d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:36:55.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.037+0000 7f4307d6d640 1 -- 172.21.15.107:0/229315361 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43000627b0 con 0x7f43000a98f0 2024-03-25T19:36:55.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.075+0000 7f42f67fc640 1 -- 172.21.15.107:0/229315361 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f42fc052300 con 0x7f43000a98f0 2024-03-25T19:36:55.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.075+0000 7f42f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:55.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.075+0000 7f42f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:55.075 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:36:55.076 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:55.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.078+0000 7f4307d6d640 10 librados: watch_flush enter 2024-03-25T19:36:55.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.078+0000 7f4307d6d640 10 librados: watch_flush exit 2024-03-25T19:36:55.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.078+0000 7f4307d6d640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-25T19:36:55.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.078+0000 7f4307d6d640 10 client.4258.objecter successfully canceled tick 2024-03-25T19:36:55.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.078+0000 7f4307d6d640 1 -- 172.21.15.107:0/229315361 >> v1:172.21.15.195:6800/2984408421 conn(0x7f42e406dfb0 legacy=0x7f42e4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:55.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.078+0000 7f4307d6d640 10 monclient: shutdown 2024-03-25T19:36:55.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.078+0000 7f4307d6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:55.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.078+0000 7f4307d6d640 1 -- 172.21.15.107:0/229315361 >> v1:172.21.15.195:6789/0 conn(0x7f43000a98f0 legacy=0x7f4300138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:55.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.079+0000 7f4307d6d640 1 -- 172.21.15.107:0/229315361 shutdown_connections 2024-03-25T19:36:55.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.079+0000 7f4307d6d640 1 -- 172.21.15.107:0/229315361 >> 172.21.15.107:0/229315361 conn(0x7f43000b7ea0 msgr2=0x7f4300074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:55.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.079+0000 7f4307d6d640 1 -- 172.21.15.107:0/229315361 shutdown_connections 2024-03-25T19:36:55.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.079+0000 7f4307d6d640 1 -- 172.21.15.107:0/229315361 wait complete. 2024-03-25T19:36:55.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:36:55.079+0000 7f4307d6d640 1 librados: shutdown 2024-03-25T19:36:55.090 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:01.896 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:02.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.006+0000 7fab2ea81640 10 monclient: get_monmap_and_config 2024-03-25T19:37:02.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.006+0000 7fab2ea81640 10 monclient: build_initial_monmap 2024-03-25T19:37:02.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.006+0000 7fab2ea81640 10 monclient: monmap: 2024-03-25T19:37:02.006 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:02.006 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:02.007 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:37:02.007654+0000 2024-03-25T19:37:02.007 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:37:02.007654+0000 2024-03-25T19:37:02.007 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:02.007 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:02.007 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:02.007 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:37:02.007 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:37:02.007 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:02.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.007+0000 7fab2ea81640 1 Processor -- start 2024-03-25T19:37:02.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.007+0000 7fab2ea81640 1 -- start start 2024-03-25T19:37:02.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.007+0000 7fab2ea81640 10 monclient: init 2024-03-25T19:37:02.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.008+0000 7fab2ea81640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:02.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.008+0000 7fab2ea81640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:02.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.008+0000 7fab2ea81640 10 monclient(hunting): picked mon.noname-b con 0x7fab280a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:02.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.008+0000 7fab2ea81640 10 monclient(hunting): picked mon.noname-c con 0x7fab2813f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:02.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.008+0000 7fab2ea81640 10 monclient(hunting): picked mon.noname-a con 0x7fab281430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:02.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.008+0000 7fab2ea81640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab28142520 con 0x7fab281430c0 2024-03-25T19:37:02.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.008+0000 7fab2ea81640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab280b28a0 con 0x7fab2813f710 2024-03-25T19:37:02.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.008+0000 7fab2ea81640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab280b2a30 con 0x7fab280a98f0 2024-03-25T19:37:02.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.008+0000 7fab2ea81640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:02.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.008+0000 7fab2ea81640 10 monclient(hunting): authenticate will time out at 4614.349163s 2024-03-25T19:37:02.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.012+0000 7fab2c81d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fab280a98f0 0x7fab280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38762/0 (socket says 172.21.15.107:38762) 2024-03-25T19:37:02.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.012+0000 7fab2c81d640 1 -- 172.21.15.107:0/2469357255 learned_addr learned my addr 172.21.15.107:0/2469357255 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:02.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.021+0000 7fab277fe640 1 -- 172.21.15.107:0/2469357255 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098415823 0 0) 0x7fab280b28a0 con 0x7fab2813f710 2024-03-25T19:37:02.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.021+0000 7fab277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:02.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.021+0000 7fab277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:02.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.021+0000 7fab277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:02.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.021+0000 7fab277fe640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:37:02.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.021+0000 7fab277fe640 1 -- 172.21.15.107:0/2469357255 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab18003520 con 0x7fab2813f710 2024-03-25T19:37:02.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.027+0000 7fab277fe640 1 -- 172.21.15.107:0/2469357255 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690792750 0 0) 0x7fab280b2a30 con 0x7fab280a98f0 2024-03-25T19:37:02.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.027+0000 7fab277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:02.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.027+0000 7fab277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:02.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.027+0000 7fab277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:02.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.027+0000 7fab277fe640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:37:02.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.027+0000 7fab277fe640 1 -- 172.21.15.107:0/2469357255 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab280b28a0 con 0x7fab280a98f0 2024-03-25T19:37:02.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.027+0000 7fab277fe640 1 -- 172.21.15.107:0/2469357255 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2234354533 0 0) 0x7fab18003520 con 0x7fab2813f710 2024-03-25T19:37:02.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 1 -- 172.21.15.107:0/2469357255 >> v1:172.21.15.107:6790/0 conn(0x7fab280a98f0 legacy=0x7fab280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:02.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 1 -- 172.21.15.107:0/2469357255 >> v1:172.21.15.107:6789/0 conn(0x7fab281430c0 legacy=0x7fab2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:02.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:02.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 1 monclient: found mon.noname-c 2024-03-25T19:37:02.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 1 -- 172.21.15.107:0/2469357255 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab280b44c0 con 0x7fab2813f710 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 1 -- 172.21.15.107:0/2469357255 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fab14001690 con 0x7fab2813f710 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab277fe640 10 monclient: dump: 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:02.029 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:02.030 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:02.030 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:02.030 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:02.030 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:02.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab2ea81640 5 monclient: authenticate success, global_id 4261 2024-03-25T19:37:02.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.028+0000 7fab2ea81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:02.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.057+0000 7fab277fe640 1 -- 172.21.15.107:0/2469357255 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab140017f0 con 0x7fab2813f710 2024-03-25T19:37:02.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:02.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab277fe640 1 -- 172.21.15.107:0/2469357255 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fab14001c40 con 0x7fab2813f710 2024-03-25T19:37:02.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab2ea81640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:02.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab2ea81640 4 set_mon_vals no callback set 2024-03-25T19:37:02.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:02.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:02.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab277fe640 10 monclient: dump: 2024-03-25T19:37:02.058 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:02.058 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:02.058 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:02.058 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:02.058 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:02.058 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:02.058 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:02.059 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:02.059 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:02.059 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:02.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab2ea81640 10 monclient: shutdown 2024-03-25T19:37:02.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab2ea81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:02.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab2ea81640 1 -- 172.21.15.107:0/2469357255 >> v1:172.21.15.195:6789/0 conn(0x7fab2813f710 legacy=0x7fab281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:02.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab2ea81640 1 -- 172.21.15.107:0/2469357255 shutdown_connections 2024-03-25T19:37:02.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab2ea81640 1 -- 172.21.15.107:0/2469357255 >> 172.21.15.107:0/2469357255 conn(0x7fab280b7ea0 msgr2=0x7fab280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:02.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab2ea81640 1 -- 172.21.15.107:0/2469357255 shutdown_connections 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.058+0000 7fab2ea81640 1 -- 172.21.15.107:0/2469357255 wait complete. 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.059+0000 7fab2ea81640 10 monclient: build_initial_monmap 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.059+0000 7fab2ea81640 10 monclient: monmap: 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:37:02.062 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.059+0000 7fab2ea81640 1 librados: starting msgr at 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.059+0000 7fab2ea81640 1 librados: starting objecter 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.060+0000 7fab2ea81640 1 Processor -- start 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.060+0000 7fab2ea81640 1 -- start start 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.060+0000 7fab2ea81640 1 librados: setting wanted keys 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.060+0000 7fab2ea81640 1 librados: calling monclient init 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.060+0000 7fab2ea81640 10 monclient: init 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.060+0000 7fab2ea81640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.060+0000 7fab2ea81640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.060+0000 7fab2ea81640 10 monclient(hunting): picked mon.noname-b con 0x7fab280a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.060+0000 7fab2ea81640 10 monclient(hunting): picked mon.noname-a con 0x7fab2813f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:02.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.061+0000 7fab2ea81640 10 monclient(hunting): picked mon.noname-c con 0x7fab281430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:02.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.061+0000 7fab2ea81640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab281d51f0 con 0x7fab2813f710 2024-03-25T19:37:02.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.061+0000 7fab2ea81640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab281d7780 con 0x7fab280a98f0 2024-03-25T19:37:02.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.061+0000 7fab2ea81640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab280aba50 con 0x7fab281430c0 2024-03-25T19:37:02.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.061+0000 7fab2ea81640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:02.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.061+0000 7fab2ea81640 10 monclient(hunting): authenticate will time out at 4614.401906s 2024-03-25T19:37:02.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.070+0000 7fab27fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fab2813f710 0x7fab28138050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47054/0 (socket says 172.21.15.107:47054) 2024-03-25T19:37:02.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.070+0000 7fab27fff640 1 -- 172.21.15.107:0/1504190564 learned_addr learned my addr 172.21.15.107:0/1504190564 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:02.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.072+0000 7fab257fa640 10 client.?.objecter ms_handle_connect 0x7fab2813f710 2024-03-25T19:37:02.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.072+0000 7fab257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:02.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.077+0000 7fab257fa640 10 client.?.objecter ms_handle_connect 0x7fab281430c0 2024-03-25T19:37:02.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.077+0000 7fab257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:02.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.080+0000 7fab257fa640 10 client.?.objecter ms_handle_connect 0x7fab280a98f0 2024-03-25T19:37:02.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.080+0000 7fab257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:02.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.087+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465971622 0 0) 0x7fab281d7780 con 0x7fab280a98f0 2024-03-25T19:37:02.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.087+0000 7fab257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:02.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.087+0000 7fab257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:02.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.087+0000 7fab257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:02.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.087+0000 7fab257fa640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:37:02.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.087+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab0c004eb0 con 0x7fab280a98f0 2024-03-25T19:37:02.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390787350 0 0) 0x7fab280aba50 con 0x7fab281430c0 2024-03-25T19:37:02.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:02.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:02.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:02.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:37:02.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab281d7780 con 0x7fab281430c0 2024-03-25T19:37:02.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704083456 0 0) 0x7fab281d51f0 con 0x7fab2813f710 2024-03-25T19:37:02.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:02.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:02.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:02.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:37:02.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.090+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab280aba50 con 0x7fab2813f710 2024-03-25T19:37:02.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.097+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2343798041 0 0) 0x7fab281d7780 con 0x7fab281430c0 2024-03-25T19:37:02.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.097+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab281d51f0 con 0x7fab281430c0 2024-03-25T19:37:02.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.097+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fab20001b00 con 0x7fab281430c0 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.097+0000 7fab257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.097+0000 7fab257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.097+0000 7fab257fa640 10 monclient(hunting): dump: 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:02.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.097+0000 7fab257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:02.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.100+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2065956810 0 0) 0x7fab0c004eb0 con 0x7fab280a98f0 2024-03-25T19:37:02.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.100+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab281d7780 con 0x7fab280a98f0 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.100+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fab1c001930 con 0x7fab280a98f0 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.100+0000 7fab257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.100+0000 7fab257fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.100+0000 7fab257fa640 10 monclient(hunting): dump: 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:02.100 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:02.101 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:02.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2060413053 0 0) 0x7fab281d51f0 con 0x7fab281430c0 2024-03-25T19:37:02.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 >> v1:172.21.15.195:6789/0 conn(0x7fab280a98f0 legacy=0x7fab281d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:02.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 >> v1:172.21.15.107:6789/0 conn(0x7fab2813f710 legacy=0x7fab28138050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab257fa640 10 monclient: _finish_hunting 0 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab257fa640 1 monclient: found mon.c 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab257fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab281d8370 con 0x7fab281430c0 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab257fa640 10 monclient: _finish_auth 0 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab257fa640 10 monclient: _check_auth_tickets 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab2ea81640 5 monclient: authenticate success, global_id 4277 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab2ea81640 10 monclient: _renew_subs 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab2ea81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab2ea81640 1 -- 172.21.15.107:0/1504190564 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab281e2e40 con 0x7fab281430c0 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab2ea81640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:02.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab2ea81640 10 monclient: _renew_subs 2024-03-25T19:37:02.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab2ea81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:02.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab2ea81640 1 -- 172.21.15.107:0/1504190564 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab281e31b0 con 0x7fab281430c0 2024-03-25T19:37:02.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.119+0000 7fab2ea81640 1 librados: init done 2024-03-25T19:37:02.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.120+0000 7fab2ea81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:02.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.120+0000 7fab2ea81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:02.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.120+0000 7fab2ea81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:02.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.120+0000 7fab2ea81640 1 -- 172.21.15.107:0/1504190564 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab280627b0 con 0x7fab281430c0 2024-03-25T19:37:02.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.464+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab20001fb0 con 0x7fab281430c0 2024-03-25T19:37:02.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.464+0000 7fab257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.464+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fab20003b10 con 0x7fab281430c0 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.464+0000 7fab257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.464+0000 7fab257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.464+0000 7fab257fa640 10 monclient: dump: 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:02.464 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:02.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.464+0000 7fab26ffd640 4 set_mon_vals no callback set 2024-03-25T19:37:02.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.465+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fab20003d30 con 0x7fab281430c0 2024-03-25T19:37:02.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.465+0000 7fab2c81d640 10 monclient: get_auth_request con 0x7fab0c06df70 auth_method 0 2024-03-25T19:37:02.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.465+0000 7fab257fa640 10 client.4277.objecter ms_handle_connect 0x7fab0c06df70 2024-03-25T19:37:02.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.471+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fab20053730 con 0x7fab281430c0 2024-03-25T19:37:02.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.471+0000 7fab257fa640 10 client.4277.objecter ms_dispatch 0x7fab28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:02.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.471+0000 7fab257fa640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:02.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.471+0000 7fab257fa640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:02.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.471+0000 7fab257fa640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-25T19:37:02.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.471+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab2004f240 con 0x7fab281430c0 2024-03-25T19:37:02.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.471+0000 7fab257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:02.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.471+0000 7fab257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:02.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.681+0000 7fab2ea81640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:02.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.681+0000 7fab2ea81640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:02.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.681+0000 7fab2ea81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:02.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.681+0000 7fab2ea81640 1 -- 172.21.15.107:0/1504190564 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab281d8500 con 0x7fab281430c0 2024-03-25T19:37:02.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.764+0000 7fab257fa640 1 -- 172.21.15.107:0/1504190564 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fab20051170 con 0x7fab281430c0 2024-03-25T19:37:02.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.764+0000 7fab257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:02.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.764+0000 7fab257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:02.764 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:37:02.764 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 10 librados: watch_flush enter 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 10 librados: watch_flush exit 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 10 client.4277.objecter successfully canceled tick 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 1 -- 172.21.15.107:0/1504190564 >> v1:172.21.15.195:6800/2984408421 conn(0x7fab0c06df70 legacy=0x7fab0c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 10 monclient: shutdown 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 1 -- 172.21.15.107:0/1504190564 >> v1:172.21.15.107:6790/0 conn(0x7fab281430c0 legacy=0x7fab28138760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 1 -- 172.21.15.107:0/1504190564 shutdown_connections 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 1 -- 172.21.15.107:0/1504190564 >> 172.21.15.107:0/1504190564 conn(0x7fab280b7ea0 msgr2=0x7fab28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 1 -- 172.21.15.107:0/1504190564 shutdown_connections 2024-03-25T19:37:02.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 1 -- 172.21.15.107:0/1504190564 wait complete. 2024-03-25T19:37:02.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:02.767+0000 7fab2ea81640 1 librados: shutdown 2024-03-25T19:37:02.778 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:04.349 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:01:34.665 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.398 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:01:34.714 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.455 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:01:34.741 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.464 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:01:34.751 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.472 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:01:34.788 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.478 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:01:34.795 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.482 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:01:34.798 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.514 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:01:34.801 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.540 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:01:34.857 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:09.585 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:09.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.699+0000 7f740d6e0640 10 monclient: get_monmap_and_config 2024-03-25T19:37:09.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.699+0000 7f740d6e0640 10 monclient: build_initial_monmap 2024-03-25T19:37:09.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.699+0000 7f740d6e0640 10 monclient: monmap: 2024-03-25T19:37:09.699 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:09.699 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:09.700 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:37:09.701003+0000 2024-03-25T19:37:09.700 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:37:09.701003+0000 2024-03-25T19:37:09.700 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:09.700 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:09.700 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:09.700 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:37:09.700 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:37:09.700 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:09.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.699+0000 7f740d6e0640 1 Processor -- start 2024-03-25T19:37:09.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 1 -- start start 2024-03-25T19:37:09.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 10 monclient: init 2024-03-25T19:37:09.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:09.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:09.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 10 monclient(hunting): picked mon.noname-a con 0x7f74080b4aa0 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:09.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 10 monclient(hunting): picked mon.noname-c con 0x7f74080b4e80 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:09.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 10 monclient(hunting): picked mon.noname-b con 0x7f74080b2140 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:09.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74080b0fe0 con 0x7f74080b4aa0 2024-03-25T19:37:09.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740814ae50 con 0x7f74080b4e80 2024-03-25T19:37:09.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74080b1230 con 0x7f74080b2140 2024-03-25T19:37:09.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:09.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.700+0000 7f740d6e0640 10 monclient(hunting): authenticate will time out at 4622.042734s 2024-03-25T19:37:09.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.711+0000 7f74067fc640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f74080b4e80 0x7f74080b1a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:45790/0 (socket says 172.21.15.107:45790) 2024-03-25T19:37:09.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.711+0000 7f74067fc640 1 -- 172.21.15.107:0/1324849772 learned_addr learned my addr 172.21.15.107:0/1324849772 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:09.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.727+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291726337 0 0) 0x7f740814ae50 con 0x7f74080b4e80 2024-03-25T19:37:09.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.727+0000 7f7405ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:09.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.727+0000 7f7405ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:09.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.727+0000 7f7405ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:09.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.727+0000 7f7405ffb640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:37:09.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.727+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f0003520 con 0x7f74080b4e80 2024-03-25T19:37:09.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.729+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469067262 0 0) 0x7f74080b0fe0 con 0x7f74080b4aa0 2024-03-25T19:37:09.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.729+0000 7f7405ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:09.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.729+0000 7f7405ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:09.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.729+0000 7f7405ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:09.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.729+0000 7f7405ffb640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:37:09.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.729+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f740814ae50 con 0x7f74080b4aa0 2024-03-25T19:37:09.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.738+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558844455 0 0) 0x7f74080b1230 con 0x7f74080b2140 2024-03-25T19:37:09.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.738+0000 7f7405ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:09.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.738+0000 7f7405ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:09.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.738+0000 7f7405ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:09.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.738+0000 7f7405ffb640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:37:09.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.738+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74080b0fe0 con 0x7f74080b2140 2024-03-25T19:37:09.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3462817676 0 0) 0x7f74080b0fe0 con 0x7f74080b2140 2024-03-25T19:37:09.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 >> v1:172.21.15.195:6789/0 conn(0x7f74080b4e80 legacy=0x7f74080b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:09.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 >> v1:172.21.15.107:6789/0 conn(0x7f74080b4aa0 legacy=0x7f740814a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:09.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 1 monclient: found mon.noname-b 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74080b0000 con 0x7f74080b2140 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7400001930 con 0x7f74080b2140 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f740d6e0640 5 monclient: authenticate success, global_id 4280 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f740d6e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f7405ffb640 10 monclient: dump: 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:09.748 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:09.749 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:09.749 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:09.749 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:09.749 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:09.749 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:09.749 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:09.749 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:09.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.746+0000 7f740d6e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:09.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.758+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7400001a90 con 0x7f74080b2140 2024-03-25T19:37:09.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.758+0000 7f7405ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:09.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.758+0000 7f7405ffb640 1 -- 172.21.15.107:0/1324849772 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7400001ee0 con 0x7f74080b2140 2024-03-25T19:37:09.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.758+0000 7f7405ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:09.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.758+0000 7f7405ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:09.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.758+0000 7f7405ffb640 10 monclient: dump: 2024-03-25T19:37:09.759 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:09.759 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:09.759 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:09.759 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:09.759 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:09.760 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:09.760 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:09.760 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:09.760 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:09.760 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:09.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.758+0000 7f740d6e0640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:09.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.758+0000 7f740d6e0640 4 set_mon_vals no callback set 2024-03-25T19:37:09.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.759+0000 7f740d6e0640 10 monclient: shutdown 2024-03-25T19:37:09.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.759+0000 7f740d6e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:09.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.759+0000 7f740d6e0640 1 -- 172.21.15.107:0/1324849772 >> v1:172.21.15.107:6790/0 conn(0x7f74080b2140 legacy=0x7f74081522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:09.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.759+0000 7f740d6e0640 1 -- 172.21.15.107:0/1324849772 shutdown_connections 2024-03-25T19:37:09.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.759+0000 7f740d6e0640 1 -- 172.21.15.107:0/1324849772 >> 172.21.15.107:0/1324849772 conn(0x7f74080617a0 msgr2=0x7f7408061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.759+0000 7f740d6e0640 1 -- 172.21.15.107:0/1324849772 shutdown_connections 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.759+0000 7f740d6e0640 1 -- 172.21.15.107:0/1324849772 wait complete. 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.760+0000 7f740d6e0640 10 monclient: build_initial_monmap 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.760+0000 7f740d6e0640 10 monclient: monmap: 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:37:09.763 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.760+0000 7f740d6e0640 1 librados: starting msgr at 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.760+0000 7f740d6e0640 1 librados: starting objecter 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.760+0000 7f740d6e0640 1 Processor -- start 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 1 -- start start 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 1 librados: setting wanted keys 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 1 librados: calling monclient init 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 10 monclient: init 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 10 monclient(hunting): picked mon.noname-b con 0x7f74080b2140 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 10 monclient(hunting): picked mon.noname-a con 0x7f74080b4aa0 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 10 monclient(hunting): picked mon.noname-c con 0x7f7408070130 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740814a0d0 con 0x7f74080b4aa0 2024-03-25T19:37:09.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7408071a10 con 0x7f74080b2140 2024-03-25T19:37:09.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740806f600 con 0x7f7408070130 2024-03-25T19:37:09.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:09.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.761+0000 7f740d6e0640 10 monclient(hunting): authenticate will time out at 4622.103692s 2024-03-25T19:37:09.766 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.766+0000 7f74077fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f7408070130 0x7f74081e7510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43272/0 (socket says 172.21.15.107:43272) 2024-03-25T19:37:09.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.766+0000 7f74077fe640 1 -- 172.21.15.107:0/2740238723 learned_addr learned my addr 172.21.15.107:0/2740238723 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:09.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.767+0000 7f73ef7fe640 10 client.?.objecter ms_handle_connect 0x7f7408070130 2024-03-25T19:37:09.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.767+0000 7f73ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:09.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.773+0000 7f73ef7fe640 10 client.?.objecter ms_handle_connect 0x7f74080b4aa0 2024-03-25T19:37:09.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.773+0000 7f73ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:09.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.774+0000 7f73ef7fe640 10 client.?.objecter ms_handle_connect 0x7f74080b2140 2024-03-25T19:37:09.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.774+0000 7f73ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:09.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.776+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604850620 0 0) 0x7f740806f600 con 0x7f7408070130 2024-03-25T19:37:09.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.776+0000 7f73ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:09.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.776+0000 7f73ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:09.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.776+0000 7f73ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:09.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.776+0000 7f73ef7fe640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:37:09.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.776+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f4004eb0 con 0x7f7408070130 2024-03-25T19:37:09.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.779+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931501519 0 0) 0x7f740814a0d0 con 0x7f74080b4aa0 2024-03-25T19:37:09.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.779+0000 7f73ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:09.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.779+0000 7f73ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:09.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.779+0000 7f73ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:09.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.779+0000 7f73ef7fe640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:37:09.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.779+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f740806f600 con 0x7f74080b4aa0 2024-03-25T19:37:09.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.789+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 324834911 0 0) 0x7f73f4004eb0 con 0x7f7408070130 2024-03-25T19:37:09.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.789+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f740814a0d0 con 0x7f7408070130 2024-03-25T19:37:09.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.789+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7400001ea0 con 0x7f7408070130 2024-03-25T19:37:09.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.789+0000 7f73ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:09.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.789+0000 7f73ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:09.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.789+0000 7f73ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.789+0000 7f73ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.790+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148124133 0 0) 0x7f7408071a10 con 0x7f74080b2140 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.790+0000 7f73ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.790+0000 7f73ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.790+0000 7f73ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:09.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.790+0000 7f73ef7fe640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:37:09.792 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.790+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f4004eb0 con 0x7f74080b2140 2024-03-25T19:37:09.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.800+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1515452251 0 0) 0x7f740806f600 con 0x7f74080b4aa0 2024-03-25T19:37:09.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.800+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7408071a10 con 0x7f74080b4aa0 2024-03-25T19:37:09.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.800+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f73fc001950 con 0x7f74080b4aa0 2024-03-25T19:37:09.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.800+0000 7f73ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:09.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.800+0000 7f73ef7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:09.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.800+0000 7f73ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:09.802 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:09.802 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:09.802 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:09.802 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:09.802 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:09.802 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:09.802 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:09.802 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:09.802 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:09.802 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:09.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.803+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3386434360 0 0) 0x7f73f4004eb0 con 0x7f74080b2140 2024-03-25T19:37:09.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.804+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f740806f600 con 0x7f74080b2140 2024-03-25T19:37:09.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.804+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f73f80018b0 con 0x7f74080b2140 2024-03-25T19:37:09.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.804+0000 7f73ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:09.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.804+0000 7f73ef7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:09.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.804+0000 7f73ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:09.805 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:09.805 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:09.805 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:09.805 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:09.805 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:09.805 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:09.805 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:09.805 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:09.805 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:09.805 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:09.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1437473732 0 0) 0x7f7408071a10 con 0x7f74080b4aa0 2024-03-25T19:37:09.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 >> v1:172.21.15.107:6790/0 conn(0x7f7408070130 legacy=0x7f74081e7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:09.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 >> v1:172.21.15.195:6789/0 conn(0x7f74080b2140 legacy=0x7f74080ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f73ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f73ef7fe640 1 monclient: found mon.a 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f73ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f73ef7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74080706e0 con 0x7f74080b4aa0 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f73ef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f73ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f73ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f740d6e0640 5 monclient: authenticate success, global_id 4281 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f740d6e0640 10 monclient: _renew_subs 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f740d6e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f740d6e0640 1 -- 172.21.15.107:0/2740238723 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7408146c80 con 0x7f74080b4aa0 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f740d6e0640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f740d6e0640 10 monclient: _renew_subs 2024-03-25T19:37:09.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f740d6e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:09.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f740d6e0640 1 -- 172.21.15.107:0/2740238723 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f740806fce0 con 0x7f74080b4aa0 2024-03-25T19:37:09.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.807+0000 7f740d6e0640 1 librados: init done 2024-03-25T19:37:09.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.808+0000 7f740d6e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:09.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.808+0000 7f740d6e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:09.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.808+0000 7f740d6e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:09.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:09.808+0000 7f740d6e0640 1 -- 172.21.15.107:0/2740238723 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7408070510 con 0x7f74080b4aa0 2024-03-25T19:37:10.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.109+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73fc001d60 con 0x7f74080b4aa0 2024-03-25T19:37:10.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.109+0000 7f73ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:10.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.109+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f73fc004980 con 0x7f74080b4aa0 2024-03-25T19:37:10.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.109+0000 7f73ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:10.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.109+0000 7f73ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:10.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.109+0000 7f73ef7fe640 10 monclient: dump: 2024-03-25T19:37:10.109 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:10.109 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:10.109 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:10.109 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:10.110 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:10.110 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:10.110 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:10.110 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:10.110 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:10.110 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:10.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.109+0000 7f74057fa640 4 set_mon_vals no callback set 2024-03-25T19:37:10.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.110+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f73fc004ba0 con 0x7f74080b4aa0 2024-03-25T19:37:10.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.111+0000 7f7406ffd640 10 monclient: get_auth_request con 0x7f73f406e0f0 auth_method 0 2024-03-25T19:37:10.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.111+0000 7f73ef7fe640 10 client.4281.objecter ms_handle_connect 0x7f73f406e0f0 2024-03-25T19:37:10.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.115+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f73fc054680 con 0x7f74080b4aa0 2024-03-25T19:37:10.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.115+0000 7f73ef7fe640 10 client.4281.objecter ms_dispatch 0x7f7408000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:10.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.115+0000 7f73ef7fe640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:10.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.115+0000 7f73ef7fe640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:10.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.115+0000 7f73ef7fe640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-25T19:37:10.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.115+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73fc050190 con 0x7f74080b4aa0 2024-03-25T19:37:10.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.115+0000 7f73ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:10.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.115+0000 7f73ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:10.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.333+0000 7f740d6e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:10.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.333+0000 7f740d6e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:10.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.333+0000 7f740d6e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:10.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.333+0000 7f740d6e0640 1 -- 172.21.15.107:0/2740238723 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7408146e10 con 0x7f74080b4aa0 2024-03-25T19:37:10.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.363+0000 7f73ef7fe640 1 -- 172.21.15.107:0/2740238723 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f7408146e10 con 0x7f74080b4aa0 2024-03-25T19:37:10.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.363+0000 7f73ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:10.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.363+0000 7f73ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:10.363 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:37:10.363 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:10.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 10 librados: watch_flush enter 2024-03-25T19:37:10.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 10 librados: watch_flush exit 2024-03-25T19:37:10.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-25T19:37:10.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 10 client.4281.objecter successfully canceled tick 2024-03-25T19:37:10.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 1 -- 172.21.15.107:0/2740238723 >> v1:172.21.15.195:6800/2984408421 conn(0x7f73f406e0f0 legacy=0x7f73f40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:10.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 10 monclient: shutdown 2024-03-25T19:37:10.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:10.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 1 -- 172.21.15.107:0/2740238723 >> v1:172.21.15.107:6789/0 conn(0x7f74080b4aa0 legacy=0x7f74081df170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:10.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 1 -- 172.21.15.107:0/2740238723 shutdown_connections 2024-03-25T19:37:10.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 1 -- 172.21.15.107:0/2740238723 >> 172.21.15.107:0/2740238723 conn(0x7f74080617a0 msgr2=0x7f7408061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:10.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 1 -- 172.21.15.107:0/2740238723 shutdown_connections 2024-03-25T19:37:10.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 1 -- 172.21.15.107:0/2740238723 wait complete. 2024-03-25T19:37:10.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:10.366+0000 7f740d6e0640 1 librados: shutdown 2024-03-25T19:37:10.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:17.184 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.300+0000 7fe89204a640 10 monclient: get_monmap_and_config 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.300+0000 7fe89204a640 10 monclient: build_initial_monmap 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.300+0000 7fe89204a640 10 monclient: monmap: 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:37:17.301449+0000 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:37:17.301449+0000 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:37:17.300 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:37:17.301 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.301+0000 7fe89204a640 1 Processor -- start 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.301+0000 7fe89204a640 1 -- start start 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.301+0000 7fe89204a640 10 monclient: init 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.302+0000 7fe89204a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.302+0000 7fe89204a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.302+0000 7fe89204a640 10 monclient(hunting): picked mon.noname-b con 0x7fe88c0705b0 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.302+0000 7fe89204a640 10 monclient(hunting): picked mon.noname-c con 0x7fe88c1402d0 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.302+0000 7fe89204a640 10 monclient(hunting): picked mon.noname-a con 0x7fe88c149ae0 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.302+0000 7fe89204a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88c0718c0 con 0x7fe88c149ae0 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.302+0000 7fe89204a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88c0714c0 con 0x7fe88c1402d0 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.302+0000 7fe89204a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88c072b50 con 0x7fe88c0705b0 2024-03-25T19:37:17.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.302+0000 7fe89204a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:17.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.302+0000 7fe89204a640 10 monclient(hunting): authenticate will time out at 4629.643032s 2024-03-25T19:37:17.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.311+0000 7fe88affd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe88c1402d0 0x7fe88c140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:35460/0 (socket says 172.21.15.107:35460) 2024-03-25T19:37:17.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.311+0000 7fe88affd640 1 -- 172.21.15.107:0/4080700295 learned_addr learned my addr 172.21.15.107:0/4080700295 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:17.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.318+0000 7fe88a7fc640 1 -- 172.21.15.107:0/4080700295 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192444723 0 0) 0x7fe88c0714c0 con 0x7fe88c1402d0 2024-03-25T19:37:17.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.318+0000 7fe88a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:17.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.318+0000 7fe88a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:17.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.318+0000 7fe88a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:17.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.318+0000 7fe88a7fc640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:37:17.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.318+0000 7fe88a7fc640 1 -- 172.21.15.107:0/4080700295 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe874003520 con 0x7fe88c1402d0 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 1 -- 172.21.15.107:0/4080700295 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1748825765 0 0) 0x7fe874003520 con 0x7fe88c1402d0 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 1 -- 172.21.15.107:0/4080700295 >> v1:172.21.15.107:6790/0 conn(0x7fe88c0705b0 legacy=0x7fe88c13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 1 -- 172.21.15.107:0/4080700295 >> v1:172.21.15.107:6789/0 conn(0x7fe88c149ae0 legacy=0x7fe88c14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 1 -- 172.21.15.107:0/4080700295 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88c13dd10 con 0x7fe88c1402d0 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 1 -- 172.21.15.107:0/4080700295 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe880001690 con 0x7fe88c1402d0 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:17.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe88a7fc640 10 monclient: dump: 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe89204a640 5 monclient: authenticate success, global_id 4273 2024-03-25T19:37:17.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.327+0000 7fe89204a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:17.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.346+0000 7fe88a7fc640 1 -- 172.21.15.107:0/4080700295 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe880001860 con 0x7fe88c1402d0 2024-03-25T19:37:17.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.346+0000 7fe88a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:17.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.346+0000 7fe88a7fc640 1 -- 172.21.15.107:0/4080700295 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe880001cb0 con 0x7fe88c1402d0 2024-03-25T19:37:17.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.346+0000 7fe89204a640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:17.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.346+0000 7fe89204a640 4 set_mon_vals no callback set 2024-03-25T19:37:17.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.346+0000 7fe88a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:17.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.346+0000 7fe88a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:17.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.346+0000 7fe88a7fc640 10 monclient: dump: 2024-03-25T19:37:17.346 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:17.346 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:17.347 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.347 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.347 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:17.347 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:17.347 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:17.347 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:17.347 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:17.347 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:17.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.347+0000 7fe89204a640 10 monclient: shutdown 2024-03-25T19:37:17.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.347+0000 7fe89204a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:17.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.347+0000 7fe89204a640 1 -- 172.21.15.107:0/4080700295 >> v1:172.21.15.195:6789/0 conn(0x7fe88c1402d0 legacy=0x7fe88c140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:17.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.347+0000 7fe89204a640 1 -- 172.21.15.107:0/4080700295 shutdown_connections 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.347+0000 7fe89204a640 1 -- 172.21.15.107:0/4080700295 >> 172.21.15.107:0/4080700295 conn(0x7fe88c0b5d10 msgr2=0x7fe88c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.347+0000 7fe89204a640 1 -- 172.21.15.107:0/4080700295 shutdown_connections 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.347+0000 7fe89204a640 1 -- 172.21.15.107:0/4080700295 wait complete. 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.348+0000 7fe89204a640 10 monclient: build_initial_monmap 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.348+0000 7fe89204a640 10 monclient: monmap: 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:17.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.348+0000 7fe89204a640 1 librados: starting msgr at 2024-03-25T19:37:17.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.348+0000 7fe89204a640 1 librados: starting objecter 2024-03-25T19:37:17.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.348+0000 7fe89204a640 1 Processor -- start 2024-03-25T19:37:17.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 1 -- start start 2024-03-25T19:37:17.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 1 librados: setting wanted keys 2024-03-25T19:37:17.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 1 librados: calling monclient init 2024-03-25T19:37:17.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 10 monclient: init 2024-03-25T19:37:17.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:17.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:17.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 10 monclient(hunting): picked mon.noname-a con 0x7fe88c0705b0 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:17.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 10 monclient(hunting): picked mon.noname-b con 0x7fe88c1402d0 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:17.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 10 monclient(hunting): picked mon.noname-c con 0x7fe88c149ae0 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:17.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88c13fca0 con 0x7fe88c0705b0 2024-03-25T19:37:17.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88c06bc90 con 0x7fe88c1402d0 2024-03-25T19:37:17.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88c06caf0 con 0x7fe88c149ae0 2024-03-25T19:37:17.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:17.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.349+0000 7fe89204a640 10 monclient(hunting): authenticate will time out at 4629.690572s 2024-03-25T19:37:17.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.368+0000 7fe88affd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe88c1402d0 0x7fe88c1d75f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:35498/0 (socket says 172.21.15.107:35498) 2024-03-25T19:37:17.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.368+0000 7fe88b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe88c0705b0 0x7fe88c143c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45578/0 (socket says 172.21.15.107:45578) 2024-03-25T19:37:17.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.368+0000 7fe88affd640 1 -- 172.21.15.107:0/2458446400 learned_addr learned my addr 172.21.15.107:0/2458446400 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:17.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.369+0000 7fe873fff640 10 client.?.objecter ms_handle_connect 0x7fe88c1402d0 2024-03-25T19:37:17.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.369+0000 7fe873fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:17.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.370+0000 7fe873fff640 10 client.?.objecter ms_handle_connect 0x7fe88c149ae0 2024-03-25T19:37:17.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.370+0000 7fe873fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:17.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.371+0000 7fe873fff640 10 client.?.objecter ms_handle_connect 0x7fe88c0705b0 2024-03-25T19:37:17.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.371+0000 7fe873fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:17.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.381+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062839509 0 0) 0x7fe88c06bc90 con 0x7fe88c1402d0 2024-03-25T19:37:17.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.381+0000 7fe873fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:17.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.381+0000 7fe873fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:17.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.381+0000 7fe873fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:17.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.381+0000 7fe873fff640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:37:17.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.381+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe878004eb0 con 0x7fe88c1402d0 2024-03-25T19:37:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184674962 0 0) 0x7fe88c06caf0 con 0x7fe88c149ae0 2024-03-25T19:37:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:37:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe88c06bc90 con 0x7fe88c149ae0 2024-03-25T19:37:17.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164468240 0 0) 0x7fe88c13fca0 con 0x7fe88c0705b0 2024-03-25T19:37:17.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:17.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:17.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:17.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:37:17.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.388+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe88c06caf0 con 0x7fe88c0705b0 2024-03-25T19:37:17.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.398+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1527219754 0 0) 0x7fe878004eb0 con 0x7fe88c1402d0 2024-03-25T19:37:17.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.398+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe88c13fca0 con 0x7fe88c1402d0 2024-03-25T19:37:17.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.398+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe880001b70 con 0x7fe88c1402d0 2024-03-25T19:37:17.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.398+0000 7fe873fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:17.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.398+0000 7fe873fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:17.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.398+0000 7fe873fff640 10 monclient(hunting): dump: 2024-03-25T19:37:17.398 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:17.398 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:17.398 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.399 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.399 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:17.399 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:17.399 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:17.399 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:17.399 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:17.399 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:17.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.398+0000 7fe873fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:17.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.422+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 850938345 0 0) 0x7fe88c06caf0 con 0x7fe88c0705b0 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.422+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe878004eb0 con 0x7fe88c0705b0 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.422+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe87c0018b0 con 0x7fe88c0705b0 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.422+0000 7fe873fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.422+0000 7fe873fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.422+0000 7fe873fff640 10 monclient(hunting): dump: 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:17.422 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:17.423 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:17.423 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:17.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.432+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3368356729 0 0) 0x7fe88c06bc90 con 0x7fe88c149ae0 2024-03-25T19:37:17.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.432+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe88c06caf0 con 0x7fe88c149ae0 2024-03-25T19:37:17.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.432+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe884001a70 con 0x7fe88c149ae0 2024-03-25T19:37:17.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.432+0000 7fe873fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:17.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.432+0000 7fe873fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:17.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.432+0000 7fe873fff640 10 monclient(hunting): dump: 2024-03-25T19:37:17.431 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:17.432 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:17.432 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.432 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.432 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:17.432 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:17.432 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:17.432 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:17.432 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:17.432 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:17.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1492093932 0 0) 0x7fe88c13fca0 con 0x7fe88c1402d0 2024-03-25T19:37:17.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 >> v1:172.21.15.107:6790/0 conn(0x7fe88c149ae0 legacy=0x7fe88c1d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:17.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 >> v1:172.21.15.107:6789/0 conn(0x7fe88c0705b0 legacy=0x7fe88c143c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:17.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe873fff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:17.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe873fff640 1 monclient: found mon.b 2024-03-25T19:37:17.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe873fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:17.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe873fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88c06cc80 con 0x7fe88c1402d0 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe873fff640 10 monclient: _finish_auth 0 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe873fff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe873fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe89204a640 5 monclient: authenticate success, global_id 4276 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe89204a640 10 monclient: _renew_subs 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe89204a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe89204a640 1 -- 172.21.15.107:0/2458446400 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88c144320 con 0x7fe88c1402d0 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe89204a640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe89204a640 10 monclient: _renew_subs 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe89204a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe89204a640 1 -- 172.21.15.107:0/2458446400 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88c148ee0 con 0x7fe88c1402d0 2024-03-25T19:37:17.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.434+0000 7fe89204a640 1 librados: init done 2024-03-25T19:37:17.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.435+0000 7fe89204a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:17.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.435+0000 7fe89204a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:17.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.435+0000 7fe89204a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:37:17.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.435+0000 7fe89204a640 1 -- 172.21.15.107:0/2458446400 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88c0627b0 con 0x7fe88c1402d0 2024-03-25T19:37:17.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.744+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe880004bf0 con 0x7fe88c1402d0 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.744+0000 7fe873fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.744+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe880004f00 con 0x7fe88c1402d0 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.744+0000 7fe873fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.744+0000 7fe873fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.744+0000 7fe873fff640 10 monclient: dump: 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:17.744 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:17.745 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:17.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.744+0000 7fe889ffb640 4 set_mon_vals no callback set 2024-03-25T19:37:17.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.745+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fe88001aed0 con 0x7fe88c1402d0 2024-03-25T19:37:17.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.746+0000 7fe88b7fe640 10 monclient: get_auth_request con 0x7fe87806e0b0 auth_method 0 2024-03-25T19:37:17.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.746+0000 7fe873fff640 10 client.4276.objecter ms_handle_connect 0x7fe87806e0b0 2024-03-25T19:37:17.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.750+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fe880054990 con 0x7fe88c1402d0 2024-03-25T19:37:17.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.750+0000 7fe873fff640 10 client.4276.objecter ms_dispatch 0x7fe88c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:17.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.750+0000 7fe873fff640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:17.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.750+0000 7fe873fff640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:17.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.750+0000 7fe873fff640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T19:37:17.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.750+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8800502d0 con 0x7fe88c1402d0 2024-03-25T19:37:17.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.750+0000 7fe873fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:17.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.750+0000 7fe873fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:17.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.998+0000 7fe89204a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:17.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.998+0000 7fe89204a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:17.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.998+0000 7fe89204a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:37:17.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:17.998+0000 7fe89204a640 1 -- 172.21.15.107:0/2458446400 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe88c13fe30 con 0x7fe88c1402d0 2024-03-25T19:37:18.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.047+0000 7fe873fff640 1 -- 172.21.15.107:0/2458446400 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fe8800523d0 con 0x7fe88c1402d0 2024-03-25T19:37:18.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.047+0000 7fe873fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:18.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.047+0000 7fe873fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:18.047 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:37:18.047 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:18.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 10 librados: watch_flush enter 2024-03-25T19:37:18.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 10 librados: watch_flush exit 2024-03-25T19:37:18.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T19:37:18.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 10 client.4276.objecter successfully canceled tick 2024-03-25T19:37:18.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 1 -- 172.21.15.107:0/2458446400 >> v1:172.21.15.195:6800/2984408421 conn(0x7fe87806e0b0 legacy=0x7fe878070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:18.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 10 monclient: shutdown 2024-03-25T19:37:18.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:18.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 1 -- 172.21.15.107:0/2458446400 >> v1:172.21.15.195:6789/0 conn(0x7fe88c1402d0 legacy=0x7fe88c1d75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:18.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 1 -- 172.21.15.107:0/2458446400 shutdown_connections 2024-03-25T19:37:18.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 1 -- 172.21.15.107:0/2458446400 >> 172.21.15.107:0/2458446400 conn(0x7fe88c0b5d10 msgr2=0x7fe88c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:18.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 1 -- 172.21.15.107:0/2458446400 shutdown_connections 2024-03-25T19:37:18.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 1 -- 172.21.15.107:0/2458446400 wait complete. 2024-03-25T19:37:18.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:18.049+0000 7fe89204a640 1 librados: shutdown 2024-03-25T19:37:18.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:24.449 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:01:54.765 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.457 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:01:54.773 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.498 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:01:54.814 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.502 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:01:54.818 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.514 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:01:54.830 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.526 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:01:54.812 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.531 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:01:54.818 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.553 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:01:54.869 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.560 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:01:54.846 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:24.866 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:24.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.982+0000 7f0be784a640 10 monclient: get_monmap_and_config 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.982+0000 7f0be784a640 10 monclient: build_initial_monmap 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.982+0000 7f0be784a640 10 monclient: monmap: 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:37:24.983226+0000 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:37:24.983226+0000 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:37:24.982 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.982+0000 7f0be784a640 1 Processor -- start 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.983+0000 7f0be784a640 1 -- start start 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.983+0000 7f0be784a640 10 monclient: init 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.983+0000 7f0be784a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.983+0000 7f0be784a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.983+0000 7f0be784a640 10 monclient(hunting): picked mon.noname-c con 0x7f0be00a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.983+0000 7f0be784a640 10 monclient(hunting): picked mon.noname-b con 0x7f0be013f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.983+0000 7f0be784a640 10 monclient(hunting): picked mon.noname-a con 0x7f0be01430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.983+0000 7f0be784a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be0142520 con 0x7f0be01430c0 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.983+0000 7f0be784a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00b28a0 con 0x7f0be00a98f0 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.984+0000 7f0be784a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00b2a30 con 0x7f0be013f710 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.984+0000 7f0be784a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:24.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.984+0000 7f0be784a640 10 monclient(hunting): authenticate will time out at 4637.324807s 2024-03-25T19:37:24.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.992+0000 7f0be4de5640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f0be013f710 0x7f0be01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51092/0 (socket says 172.21.15.107:51092) 2024-03-25T19:37:24.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:24.992+0000 7f0be4de5640 1 -- 172.21.15.107:0/629818970 learned_addr learned my addr 172.21.15.107:0/629818970 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:25.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.002+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31949867 0 0) 0x7f0be0142520 con 0x7f0be01430c0 2024-03-25T19:37:25.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.002+0000 7f0bd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:25.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.002+0000 7f0bd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:25.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.002+0000 7f0bd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:25.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.002+0000 7f0bd7fff640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:37:25.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.002+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bcc003520 con 0x7f0be01430c0 2024-03-25T19:37:25.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.006+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141343499 0 0) 0x7f0be00b2a30 con 0x7f0be013f710 2024-03-25T19:37:25.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.006+0000 7f0bd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:25.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.006+0000 7f0bd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:25.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.006+0000 7f0bd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:25.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.006+0000 7f0bd7fff640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:37:25.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.006+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be0142520 con 0x7f0be013f710 2024-03-25T19:37:25.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.010+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767733637 0 0) 0x7f0be00b28a0 con 0x7f0be00a98f0 2024-03-25T19:37:25.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.010+0000 7f0bd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:25.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.010+0000 7f0bd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:25.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.010+0000 7f0bd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:25.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.010+0000 7f0bd7fff640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:37:25.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.010+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be00b2a30 con 0x7f0be00a98f0 2024-03-25T19:37:25.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2913785337 0 0) 0x7f0bcc003520 con 0x7f0be01430c0 2024-03-25T19:37:25.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 >> v1:172.21.15.107:6790/0 conn(0x7f0be013f710 legacy=0x7f0be01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:25.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 >> v1:172.21.15.195:6789/0 conn(0x7f0be00a98f0 legacy=0x7f0be00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:25.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:25.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 1 monclient: found mon.noname-a 2024-03-25T19:37:25.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:25.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be00b44c0 con 0x7f0be01430c0 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 10 monclient: _finish_auth 0 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f0bd8001a10 con 0x7f0be01430c0 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0bd7fff640 10 monclient: dump: 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:25.017 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:25.018 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:25.018 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:25.018 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:25.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0be784a640 5 monclient: authenticate success, global_id 4287 2024-03-25T19:37:25.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.016+0000 7f0be784a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:25.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.025+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bd8001b70 con 0x7f0be01430c0 2024-03-25T19:37:25.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.025+0000 7f0bd7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:25.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.025+0000 7f0bd7fff640 1 -- 172.21.15.107:0/629818970 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f0bd8001f80 con 0x7f0be01430c0 2024-03-25T19:37:25.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.025+0000 7f0bd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:25.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.025+0000 7f0bd7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:25.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.025+0000 7f0bd7fff640 10 monclient: dump: 2024-03-25T19:37:25.025 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:25.025 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:25.025 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.025 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.025 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:25.026 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:25.026 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:25.026 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:25.026 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:25.026 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:25.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.025+0000 7f0be784a640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:25.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.025+0000 7f0be784a640 4 set_mon_vals no callback set 2024-03-25T19:37:25.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.026+0000 7f0be784a640 10 monclient: shutdown 2024-03-25T19:37:25.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.026+0000 7f0be784a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:25.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.026+0000 7f0be784a640 1 -- 172.21.15.107:0/629818970 >> v1:172.21.15.107:6789/0 conn(0x7f0be01430c0 legacy=0x7f0be014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:25.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.026+0000 7f0be784a640 1 -- 172.21.15.107:0/629818970 shutdown_connections 2024-03-25T19:37:25.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.026+0000 7f0be784a640 1 -- 172.21.15.107:0/629818970 >> 172.21.15.107:0/629818970 conn(0x7f0be00b7ea0 msgr2=0x7f0be00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:25.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.026+0000 7f0be784a640 1 -- 172.21.15.107:0/629818970 shutdown_connections 2024-03-25T19:37:25.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.026+0000 7f0be784a640 1 -- 172.21.15.107:0/629818970 wait complete. 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.027+0000 7f0be784a640 10 monclient: build_initial_monmap 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.027+0000 7f0be784a640 10 monclient: monmap: 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.027+0000 7f0be784a640 1 librados: starting msgr at 2024-03-25T19:37:25.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.027+0000 7f0be784a640 1 librados: starting objecter 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.027+0000 7f0be784a640 1 Processor -- start 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.027+0000 7f0be784a640 1 -- start start 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.027+0000 7f0be784a640 1 librados: setting wanted keys 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.027+0000 7f0be784a640 1 librados: calling monclient init 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.027+0000 7f0be784a640 10 monclient: init 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.028+0000 7f0be784a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.028+0000 7f0be784a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.028+0000 7f0be784a640 10 monclient(hunting): picked mon.noname-a con 0x7f0be00a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.028+0000 7f0be784a640 10 monclient(hunting): picked mon.noname-b con 0x7f0be013f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.028+0000 7f0be784a640 10 monclient(hunting): picked mon.noname-c con 0x7f0be01430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.028+0000 7f0be784a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be0139830 con 0x7f0be00a98f0 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.028+0000 7f0be784a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be01db7d0 con 0x7f0be013f710 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.028+0000 7f0be784a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be01db410 con 0x7f0be01430c0 2024-03-25T19:37:25.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.028+0000 7f0be784a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:25.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.028+0000 7f0be784a640 10 monclient(hunting): authenticate will time out at 4637.369327s 2024-03-25T19:37:25.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.038+0000 7f0be4de5640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f0be013f710 0x7f0be00b0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:33918/0 (socket says 172.21.15.107:33918) 2024-03-25T19:37:25.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.038+0000 7f0be4de5640 1 -- 172.21.15.107:0/2663273405 learned_addr learned my addr 172.21.15.107:0/2663273405 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:25.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.039+0000 7f0bd5ffb640 10 client.?.objecter ms_handle_connect 0x7f0be013f710 2024-03-25T19:37:25.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.039+0000 7f0bd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:25.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.040+0000 7f0bd5ffb640 10 client.?.objecter ms_handle_connect 0x7f0be01430c0 2024-03-25T19:37:25.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.040+0000 7f0bd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:25.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.043+0000 7f0bd5ffb640 10 client.?.objecter ms_handle_connect 0x7f0be00a98f0 2024-03-25T19:37:25.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.043+0000 7f0bd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:25.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.058+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390752708 0 0) 0x7f0be0139830 con 0x7f0be00a98f0 2024-03-25T19:37:25.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.058+0000 7f0bd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:25.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.058+0000 7f0bd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:25.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.058+0000 7f0bd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:25.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.058+0000 7f0bd5ffb640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:37:25.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.058+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc4004eb0 con 0x7f0be00a98f0 2024-03-25T19:37:25.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.063+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031770175 0 0) 0x7f0be01db7d0 con 0x7f0be013f710 2024-03-25T19:37:25.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.063+0000 7f0bd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:25.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.063+0000 7f0bd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:25.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.063+0000 7f0bd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:25.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.063+0000 7f0bd5ffb640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:37:25.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.063+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be0139830 con 0x7f0be013f710 2024-03-25T19:37:25.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.065+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745681522 0 0) 0x7f0be01db410 con 0x7f0be01430c0 2024-03-25T19:37:25.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.065+0000 7f0bd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:25.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.065+0000 7f0bd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:25.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.065+0000 7f0bd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:25.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.065+0000 7f0bd5ffb640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:37:25.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.065+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be01db7d0 con 0x7f0be01430c0 2024-03-25T19:37:25.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.147+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3321067453 0 0) 0x7f0be01db7d0 con 0x7f0be01430c0 2024-03-25T19:37:25.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.147+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0be01db410 con 0x7f0be01430c0 2024-03-25T19:37:25.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.147+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f0bd8001e80 con 0x7f0be01430c0 2024-03-25T19:37:25.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.147+0000 7f0bd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:25.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.147+0000 7f0bd5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:25.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.147+0000 7f0bd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:37:25.147 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:25.148 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:25.148 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.148 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.148 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:25.148 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:25.148 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:25.148 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:25.148 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:25.148 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:25.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.147+0000 7f0bd5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:25.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.158+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2301232313 0 0) 0x7f0bc4004eb0 con 0x7f0be00a98f0 2024-03-25T19:37:25.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.158+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0be01db7d0 con 0x7f0be00a98f0 2024-03-25T19:37:25.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.158+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f0bdc0018f0 con 0x7f0be00a98f0 2024-03-25T19:37:25.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.158+0000 7f0bd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:25.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.158+0000 7f0bd5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:25.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.158+0000 7f0bd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:37:25.158 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:25.158 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:25.158 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.158 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.159 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:25.159 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:25.159 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:25.159 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:25.159 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:25.159 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.163+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1438620139 0 0) 0x7f0be0139830 con 0x7f0be013f710 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.163+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bc4004eb0 con 0x7f0be013f710 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.163+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f0bd0001970 con 0x7f0be013f710 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.163+0000 7f0bd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.163+0000 7f0bd5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.163+0000 7f0bd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:25.163 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:25.164 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:25.164 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:25.164 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:25.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2001634191 0 0) 0x7f0be01db410 con 0x7f0be01430c0 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 >> v1:172.21.15.195:6789/0 conn(0x7f0be013f710 legacy=0x7f0be00b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 >> v1:172.21.15.107:6789/0 conn(0x7f0be00a98f0 legacy=0x7f0be01dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0bd5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0bd5ffb640 1 monclient: found mon.c 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0bd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0bd5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be01db5a0 con 0x7f0be01430c0 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0bd5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0bd5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0bd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0be784a640 5 monclient: authenticate success, global_id 4295 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0be784a640 10 monclient: _renew_subs 2024-03-25T19:37:25.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0be784a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:25.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.164+0000 7f0be784a640 1 -- 172.21.15.107:0/2663273405 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be00b1480 con 0x7f0be01430c0 2024-03-25T19:37:25.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.165+0000 7f0be784a640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:25.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.165+0000 7f0be784a640 10 monclient: _renew_subs 2024-03-25T19:37:25.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.165+0000 7f0be784a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:25.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.165+0000 7f0be784a640 1 -- 172.21.15.107:0/2663273405 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be00b0930 con 0x7f0be01430c0 2024-03-25T19:37:25.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.165+0000 7f0be784a640 1 librados: init done 2024-03-25T19:37:25.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.166+0000 7f0be784a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:25.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.166+0000 7f0be784a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:25.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.166+0000 7f0be784a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:25.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.166+0000 7f0be784a640 1 -- 172.21.15.107:0/2663273405 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be00627b0 con 0x7f0be01430c0 2024-03-25T19:37:25.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.487+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bd8001560 con 0x7f0be01430c0 2024-03-25T19:37:25.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.487+0000 7f0bd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:25.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.487+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f0bd8003c20 con 0x7f0be01430c0 2024-03-25T19:37:25.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.487+0000 7f0bd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:25.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.487+0000 7f0bd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:25.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.487+0000 7f0bd5ffb640 10 monclient: dump: 2024-03-25T19:37:25.487 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:25.487 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:25.487 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.487 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:25.488 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:25.488 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:25.488 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:25.488 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:25.488 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:25.488 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:25.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.487+0000 7f0bd77fe640 4 set_mon_vals no callback set 2024-03-25T19:37:25.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.488+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f0bd8019bf0 con 0x7f0be01430c0 2024-03-25T19:37:25.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.488+0000 7f0be55e6640 10 monclient: get_auth_request con 0x7f0bc406e0f0 auth_method 0 2024-03-25T19:37:25.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.489+0000 7f0bd5ffb640 10 client.4295.objecter ms_handle_connect 0x7f0bc406e0f0 2024-03-25T19:37:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.493+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f0bd8053810 con 0x7f0be01430c0 2024-03-25T19:37:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.493+0000 7f0bd5ffb640 10 client.4295.objecter ms_dispatch 0x7f0be0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.493+0000 7f0bd5ffb640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.493+0000 7f0bd5ffb640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.493+0000 7f0bd5ffb640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-25T19:37:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.493+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0bd804f320 con 0x7f0be01430c0 2024-03-25T19:37:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.493+0000 7f0bd5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:25.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.493+0000 7f0bd5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:25.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.741+0000 7f0be784a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:25.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.741+0000 7f0be784a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:25.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.741+0000 7f0be784a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:25.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.741+0000 7f0be784a640 1 -- 172.21.15.107:0/2663273405 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0be0069f40 con 0x7f0be01430c0 2024-03-25T19:37:25.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.776+0000 7f0bd5ffb640 1 -- 172.21.15.107:0/2663273405 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f0bd8051250 con 0x7f0be01430c0 2024-03-25T19:37:25.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.777+0000 7f0bd5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:25.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.777+0000 7f0bd5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:25.777 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:37:25.777 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:25.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 10 librados: watch_flush enter 2024-03-25T19:37:25.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 10 librados: watch_flush exit 2024-03-25T19:37:25.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-25T19:37:25.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 10 client.4295.objecter successfully canceled tick 2024-03-25T19:37:25.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 1 -- 172.21.15.107:0/2663273405 >> v1:172.21.15.195:6800/2984408421 conn(0x7f0bc406e0f0 legacy=0x7f0bc40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:25.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 10 monclient: shutdown 2024-03-25T19:37:25.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:25.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 1 -- 172.21.15.107:0/2663273405 >> v1:172.21.15.107:6790/0 conn(0x7f0be01430c0 legacy=0x7f0be01d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:25.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 1 -- 172.21.15.107:0/2663273405 shutdown_connections 2024-03-25T19:37:25.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 1 -- 172.21.15.107:0/2663273405 >> 172.21.15.107:0/2663273405 conn(0x7f0be00b7ea0 msgr2=0x7f0be0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:25.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 1 -- 172.21.15.107:0/2663273405 shutdown_connections 2024-03-25T19:37:25.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 1 -- 172.21.15.107:0/2663273405 wait complete. 2024-03-25T19:37:25.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:25.779+0000 7f0be784a640 1 librados: shutdown 2024-03-25T19:37:25.791 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:32.599 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:32.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.716+0000 7fca6c2f6640 10 monclient: get_monmap_and_config 2024-03-25T19:37:32.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.716+0000 7fca6c2f6640 10 monclient: build_initial_monmap 2024-03-25T19:37:32.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.716+0000 7fca6c2f6640 10 monclient: monmap: 2024-03-25T19:37:32.716 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:32.716 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:32.716 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:37:32.717678+0000 2024-03-25T19:37:32.716 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:37:32.717678+0000 2024-03-25T19:37:32.716 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:32.716 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:32.716 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:32.717 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:37:32.717 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:37:32.717 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:32.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.717+0000 7fca6c2f6640 1 Processor -- start 2024-03-25T19:37:32.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.717+0000 7fca6c2f6640 1 -- start start 2024-03-25T19:37:32.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.717+0000 7fca6c2f6640 10 monclient: init 2024-03-25T19:37:32.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.718+0000 7fca6c2f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:32.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.718+0000 7fca6c2f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:32.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.718+0000 7fca6c2f6640 10 monclient(hunting): picked mon.noname-b con 0x7fca640b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:32.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.718+0000 7fca6c2f6640 10 monclient(hunting): picked mon.noname-c con 0x7fca640b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:32.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.718+0000 7fca6c2f6640 10 monclient(hunting): picked mon.noname-a con 0x7fca6414be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:32.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.718+0000 7fca6c2f6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640af940 con 0x7fca6414be90 2024-03-25T19:37:32.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.718+0000 7fca6c2f6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640b0a70 con 0x7fca640b1730 2024-03-25T19:37:32.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.718+0000 7fca6c2f6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640b0d50 con 0x7fca640b4d60 2024-03-25T19:37:32.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.718+0000 7fca6c2f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:32.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.718+0000 7fca6c2f6640 10 monclient(hunting): authenticate will time out at 4645.059178s 2024-03-25T19:37:32.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.729+0000 7fca6a092640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fca640b4d60 0x7fca6413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:59722/0 (socket says 172.21.15.107:59722) 2024-03-25T19:37:32.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.729+0000 7fca6a092640 1 -- 172.21.15.107:0/838685065 learned_addr learned my addr 172.21.15.107:0/838685065 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:32.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.738+0000 7fca69090640 1 -- 172.21.15.107:0/838685065 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476269923 0 0) 0x7fca640b0d50 con 0x7fca640b4d60 2024-03-25T19:37:32.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.738+0000 7fca69090640 10 monclient(hunting): _negotiate 2024-03-25T19:37:32.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.738+0000 7fca69090640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:32.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.738+0000 7fca69090640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:32.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.738+0000 7fca69090640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:37:32.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.738+0000 7fca69090640 1 -- 172.21.15.107:0/838685065 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca50003520 con 0x7fca640b4d60 2024-03-25T19:37:32.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 1 -- 172.21.15.107:0/838685065 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2728406470 0 0) 0x7fca50003520 con 0x7fca640b4d60 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 1 -- 172.21.15.107:0/838685065 >> v1:172.21.15.195:6789/0 conn(0x7fca640b1730 legacy=0x7fca640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 1 -- 172.21.15.107:0/838685065 >> v1:172.21.15.107:6789/0 conn(0x7fca6414be90 legacy=0x7fca6414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 10 monclient: _finish_hunting 0 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 1 monclient: found mon.noname-b 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 1 -- 172.21.15.107:0/838685065 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca640b1030 con 0x7fca640b4d60 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 10 monclient: _finish_auth 0 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 10 monclient: _check_auth_tickets 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 1 -- 172.21.15.107:0/838685065 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fca60001a10 con 0x7fca640b4d60 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca6c2f6640 5 monclient: authenticate success, global_id 4298 2024-03-25T19:37:32.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca6c2f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca69090640 10 monclient: dump: 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:32.749 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:32.750 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:32.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.748+0000 7fca6c2f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:32.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.758+0000 7fca69090640 1 -- 172.21.15.107:0/838685065 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca60001b70 con 0x7fca640b4d60 2024-03-25T19:37:32.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.758+0000 7fca69090640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:32.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.758+0000 7fca69090640 1 -- 172.21.15.107:0/838685065 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fca60001f80 con 0x7fca640b4d60 2024-03-25T19:37:32.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.758+0000 7fca6c2f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:32.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.758+0000 7fca6c2f6640 4 set_mon_vals no callback set 2024-03-25T19:37:32.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.758+0000 7fca69090640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:32.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.758+0000 7fca69090640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:32.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.758+0000 7fca69090640 10 monclient: dump: 2024-03-25T19:37:32.759 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:32.759 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:32.759 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:32.759 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:32.759 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:32.759 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:32.759 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:32.759 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:32.759 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:32.759 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:32.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.759+0000 7fca6c2f6640 10 monclient: shutdown 2024-03-25T19:37:32.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.759+0000 7fca6c2f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:32.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.759+0000 7fca6c2f6640 1 -- 172.21.15.107:0/838685065 >> v1:172.21.15.107:6790/0 conn(0x7fca640b4d60 legacy=0x7fca6413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:32.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.759+0000 7fca6c2f6640 1 -- 172.21.15.107:0/838685065 shutdown_connections 2024-03-25T19:37:32.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.759+0000 7fca6c2f6640 1 -- 172.21.15.107:0/838685065 >> 172.21.15.107:0/838685065 conn(0x7fca640617a0 msgr2=0x7fca64061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:32.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.759+0000 7fca6c2f6640 1 -- 172.21.15.107:0/838685065 shutdown_connections 2024-03-25T19:37:32.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.759+0000 7fca6c2f6640 1 -- 172.21.15.107:0/838685065 wait complete. 2024-03-25T19:37:32.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.760+0000 7fca6c2f6640 10 monclient: build_initial_monmap 2024-03-25T19:37:32.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.760+0000 7fca6c2f6640 10 monclient: monmap: 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.760+0000 7fca6c2f6640 1 librados: starting msgr at 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.760+0000 7fca6c2f6640 1 librados: starting objecter 2024-03-25T19:37:32.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.761+0000 7fca6c2f6640 1 Processor -- start 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.761+0000 7fca6c2f6640 1 -- start start 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.761+0000 7fca6c2f6640 1 librados: setting wanted keys 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.761+0000 7fca6c2f6640 1 librados: calling monclient init 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.761+0000 7fca6c2f6640 10 monclient: init 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.761+0000 7fca6c2f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.761+0000 7fca6c2f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.761+0000 7fca6c2f6640 10 monclient(hunting): picked mon.noname-a con 0x7fca640b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.761+0000 7fca6c2f6640 10 monclient(hunting): picked mon.noname-b con 0x7fca640b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.761+0000 7fca6c2f6640 10 monclient(hunting): picked mon.noname-c con 0x7fca6414be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.761+0000 7fca6c2f6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca64137f30 con 0x7fca640b1730 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.762+0000 7fca6c2f6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca64131720 con 0x7fca640b4d60 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.762+0000 7fca6c2f6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca641dba00 con 0x7fca6414be90 2024-03-25T19:37:32.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.762+0000 7fca6c2f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:32.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.762+0000 7fca6c2f6640 10 monclient(hunting): authenticate will time out at 4645.102812s 2024-03-25T19:37:32.772 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.772+0000 7fca6a893640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fca6414be90 0x7fca641db2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:59736/0 (socket says 172.21.15.107:59736) 2024-03-25T19:37:32.772 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.772+0000 7fca6a893640 1 -- 172.21.15.107:0/2632504543 learned_addr learned my addr 172.21.15.107:0/2632504543 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:32.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.780+0000 7fca5affd640 10 client.?.objecter ms_handle_connect 0x7fca640b4d60 2024-03-25T19:37:32.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.780+0000 7fca5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:32.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.787+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305799330 0 0) 0x7fca64131720 con 0x7fca640b4d60 2024-03-25T19:37:32.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.787+0000 7fca5affd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:32.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.787+0000 7fca5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:32.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.787+0000 7fca5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:32.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.787+0000 7fca5affd640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:37:32.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.787+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca48004eb0 con 0x7fca640b4d60 2024-03-25T19:37:32.792 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.793+0000 7fca5affd640 10 client.?.objecter ms_handle_connect 0x7fca640b1730 2024-03-25T19:37:32.792 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.793+0000 7fca5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:32.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.793+0000 7fca5affd640 10 client.?.objecter ms_handle_connect 0x7fca6414be90 2024-03-25T19:37:32.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.793+0000 7fca5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:32.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.796+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3120824858 0 0) 0x7fca48004eb0 con 0x7fca640b4d60 2024-03-25T19:37:32.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.796+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca64131720 con 0x7fca640b4d60 2024-03-25T19:37:32.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.796+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fca540018c0 con 0x7fca640b4d60 2024-03-25T19:37:32.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.796+0000 7fca5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:32.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.796+0000 7fca5affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:32.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.796+0000 7fca5affd640 10 monclient(hunting): dump: 2024-03-25T19:37:32.796 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:32.796 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:32.796 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:32.797 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:32.797 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:32.797 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:32.797 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:32.797 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:32.797 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:32.797 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:32.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.796+0000 7fca5affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:32.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.800+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177442274 0 0) 0x7fca641dba00 con 0x7fca6414be90 2024-03-25T19:37:32.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.800+0000 7fca5affd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:32.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.800+0000 7fca5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:32.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.800+0000 7fca5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:32.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.800+0000 7fca5affd640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:37:32.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.800+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca48004eb0 con 0x7fca6414be90 2024-03-25T19:37:32.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1777254522 0 0) 0x7fca64131720 con 0x7fca640b4d60 2024-03-25T19:37:32.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 >> v1:172.21.15.107:6790/0 conn(0x7fca6414be90 legacy=0x7fca641db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:32.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 >> v1:172.21.15.107:6789/0 conn(0x7fca640b1730 legacy=0x7fca64137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:32.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca5affd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:32.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca5affd640 1 monclient: found mon.b 2024-03-25T19:37:32.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:32.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca5affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:37:32.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca641dbb90 con 0x7fca640b4d60 2024-03-25T19:37:32.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca5affd640 10 monclient: _finish_auth 0 2024-03-25T19:37:32.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca5affd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:32.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:32.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca6c2f6640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:37:32.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca6c2f6640 10 monclient: _renew_subs 2024-03-25T19:37:32.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca6c2f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:37:32.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca6c2f6640 1 -- 172.21.15.107:0/2632504543 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca64143b60 con 0x7fca640b4d60 2024-03-25T19:37:32.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca6c2f6640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:32.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca6c2f6640 10 monclient: _renew_subs 2024-03-25T19:37:32.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca6c2f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:37:32.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca6c2f6640 1 -- 172.21.15.107:0/2632504543 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca64136dd0 con 0x7fca640b4d60 2024-03-25T19:37:32.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.807+0000 7fca6c2f6640 1 librados: init done 2024-03-25T19:37:32.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.808+0000 7fca6c2f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:32.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.808+0000 7fca6c2f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:32.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.808+0000 7fca6c2f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:37:32.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:32.808+0000 7fca6c2f6640 1 -- 172.21.15.107:0/2632504543 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca641370d0 con 0x7fca640b4d60 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.087+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca540035f0 con 0x7fca640b4d60 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.087+0000 7fca5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.087+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fca54003820 con 0x7fca640b4d60 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.087+0000 7fca5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.087+0000 7fca5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.087+0000 7fca5affd640 10 monclient: dump: 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:33.087 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:33.088 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:33.088 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:33.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.087+0000 7fca6888f640 4 set_mon_vals no callback set 2024-03-25T19:37:33.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.088+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fca54003980 con 0x7fca640b4d60 2024-03-25T19:37:33.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.089+0000 7fca6a092640 10 monclient: get_auth_request con 0x7fca4806dce0 auth_method 0 2024-03-25T19:37:33.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.089+0000 7fca5affd640 10 client.4288.objecter ms_handle_connect 0x7fca4806dce0 2024-03-25T19:37:33.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.094+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fca54053490 con 0x7fca640b4d60 2024-03-25T19:37:33.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.094+0000 7fca5affd640 10 client.4288.objecter ms_dispatch 0x7fca64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:33.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.094+0000 7fca5affd640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:33.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.094+0000 7fca5affd640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:33.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.094+0000 7fca5affd640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T19:37:33.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.094+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca5404efa0 con 0x7fca640b4d60 2024-03-25T19:37:33.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.094+0000 7fca5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:33.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.094+0000 7fca5affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:33.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.324+0000 7fca6c2f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:33.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.324+0000 7fca6c2f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:33.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.324+0000 7fca6c2f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:37:33.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.324+0000 7fca6c2f6640 1 -- 172.21.15.107:0/2632504543 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca64137370 con 0x7fca640b4d60 2024-03-25T19:37:33.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.361+0000 7fca5affd640 1 -- 172.21.15.107:0/2632504543 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fca64137370 con 0x7fca640b4d60 2024-03-25T19:37:33.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.361+0000 7fca5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:33.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.361+0000 7fca5affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:33.361 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:37:33.361 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:33.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.363+0000 7fca6c2f6640 10 librados: watch_flush enter 2024-03-25T19:37:33.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.363+0000 7fca6c2f6640 10 librados: watch_flush exit 2024-03-25T19:37:33.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.363+0000 7fca6c2f6640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T19:37:33.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.363+0000 7fca6c2f6640 10 client.4288.objecter successfully canceled tick 2024-03-25T19:37:33.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.363+0000 7fca6c2f6640 1 -- 172.21.15.107:0/2632504543 >> v1:172.21.15.195:6800/2984408421 conn(0x7fca4806dce0 legacy=0x7fca480701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:33.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.363+0000 7fca6c2f6640 10 monclient: shutdown 2024-03-25T19:37:33.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.363+0000 7fca6c2f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:33.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.363+0000 7fca6c2f6640 1 -- 172.21.15.107:0/2632504543 >> v1:172.21.15.195:6789/0 conn(0x7fca640b4d60 legacy=0x7fca64131010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:33.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.364+0000 7fca6c2f6640 1 -- 172.21.15.107:0/2632504543 shutdown_connections 2024-03-25T19:37:33.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.364+0000 7fca6c2f6640 1 -- 172.21.15.107:0/2632504543 >> 172.21.15.107:0/2632504543 conn(0x7fca640617a0 msgr2=0x7fca64061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:33.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.364+0000 7fca6c2f6640 1 -- 172.21.15.107:0/2632504543 shutdown_connections 2024-03-25T19:37:33.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.364+0000 7fca6c2f6640 1 -- 172.21.15.107:0/2632504543 wait complete. 2024-03-25T19:37:33.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:33.364+0000 7fca6c2f6640 1 librados: shutdown 2024-03-25T19:37:33.374 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:40.180 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:40.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.288+0000 7f1bf8d9d640 10 monclient: get_monmap_and_config 2024-03-25T19:37:40.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.288+0000 7f1bf8d9d640 10 monclient: build_initial_monmap 2024-03-25T19:37:40.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.288+0000 7f1bf8d9d640 10 monclient: monmap: 2024-03-25T19:37:40.288 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:40.288 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:40.288 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:37:40.289732+0000 2024-03-25T19:37:40.288 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:37:40.289732+0000 2024-03-25T19:37:40.289 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:40.289 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:40.289 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:40.289 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:37:40.289 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:37:40.289 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:40.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.289+0000 7f1bf8d9d640 1 Processor -- start 2024-03-25T19:37:40.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.289+0000 7f1bf8d9d640 1 -- start start 2024-03-25T19:37:40.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.289+0000 7f1bf8d9d640 10 monclient: init 2024-03-25T19:37:40.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.290+0000 7f1bf8d9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:40.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.290+0000 7f1bf8d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:40.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.290+0000 7f1bf8d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf40a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:40.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.290+0000 7f1bf8d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:40.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.290+0000 7f1bf8d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf41430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:40.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.290+0000 7f1bf8d9d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf4142520 con 0x7f1bf40a98f0 2024-03-25T19:37:40.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.290+0000 7f1bf8d9d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf40b28a0 con 0x7f1bf413f710 2024-03-25T19:37:40.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.290+0000 7f1bf8d9d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf40b2a30 con 0x7f1bf41430c0 2024-03-25T19:37:40.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.290+0000 7f1bf8d9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:40.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.290+0000 7f1bf8d9d640 10 monclient(hunting): authenticate will time out at 4652.631452s 2024-03-25T19:37:40.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.297+0000 7f1bf259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1bf40a98f0 0x7f1bf40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38976/0 (socket says 172.21.15.107:38976) 2024-03-25T19:37:40.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.297+0000 7f1bf259c640 1 -- 172.21.15.107:0/3692103765 learned_addr learned my addr 172.21.15.107:0/3692103765 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:40.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.304+0000 7f1bf159a640 1 -- 172.21.15.107:0/3692103765 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694860106 0 0) 0x7f1bf4142520 con 0x7f1bf40a98f0 2024-03-25T19:37:40.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.304+0000 7f1bf159a640 10 monclient(hunting): _negotiate 2024-03-25T19:37:40.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.304+0000 7f1bf159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:40.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.304+0000 7f1bf159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:40.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.304+0000 7f1bf159a640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:37:40.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.304+0000 7f1bf159a640 1 -- 172.21.15.107:0/3692103765 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bdc003520 con 0x7f1bf40a98f0 2024-03-25T19:37:40.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.307+0000 7f1bf159a640 1 -- 172.21.15.107:0/3692103765 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337780211 0 0) 0x7f1bf40b28a0 con 0x7f1bf413f710 2024-03-25T19:37:40.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.307+0000 7f1bf159a640 10 monclient(hunting): _negotiate 2024-03-25T19:37:40.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.307+0000 7f1bf159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:40.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.307+0000 7f1bf159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:40.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.307+0000 7f1bf159a640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:37:40.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.307+0000 7f1bf159a640 1 -- 172.21.15.107:0/3692103765 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf4142520 con 0x7f1bf413f710 2024-03-25T19:37:40.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 1 -- 172.21.15.107:0/3692103765 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3818283307 0 0) 0x7f1bf4142520 con 0x7f1bf413f710 2024-03-25T19:37:40.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 1 -- 172.21.15.107:0/3692103765 >> v1:172.21.15.107:6790/0 conn(0x7f1bf41430c0 legacy=0x7f1bf414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 1 -- 172.21.15.107:0/3692103765 >> v1:172.21.15.107:6789/0 conn(0x7f1bf40a98f0 legacy=0x7f1bf40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 10 monclient: _finish_hunting 0 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 1 monclient: found mon.noname-c 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 1 -- 172.21.15.107:0/3692103765 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf40b44c0 con 0x7f1bf413f710 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 10 monclient: _finish_auth 0 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 10 monclient: _check_auth_tickets 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 1 -- 172.21.15.107:0/3692103765 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1be8001670 con 0x7f1bf413f710 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf8d9d640 5 monclient: authenticate success, global_id 4291 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf8d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf159a640 10 monclient: dump: 2024-03-25T19:37:40.313 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:40.314 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:40.314 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:40.314 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:40.314 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:40.314 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:40.314 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:40.314 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:40.314 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:40.314 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:40.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.313+0000 7f1bf8d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:40.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.327+0000 7f1bf159a640 1 -- 172.21.15.107:0/3692103765 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1be80017d0 con 0x7f1bf413f710 2024-03-25T19:37:40.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.327+0000 7f1bf159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:40.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.327+0000 7f1bf159a640 1 -- 172.21.15.107:0/3692103765 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1be8001c20 con 0x7f1bf413f710 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.327+0000 7f1bf8d9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.327+0000 7f1bf8d9d640 4 set_mon_vals no callback set 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.327+0000 7f1bf159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.327+0000 7f1bf159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.327+0000 7f1bf159a640 10 monclient: dump: 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:40.327 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.328+0000 7f1bf8d9d640 10 monclient: shutdown 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.328+0000 7f1bf8d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.328+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/3692103765 >> v1:172.21.15.195:6789/0 conn(0x7f1bf413f710 legacy=0x7f1bf41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.328+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/3692103765 shutdown_connections 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.328+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/3692103765 >> 172.21.15.107:0/3692103765 conn(0x7f1bf40b7ea0 msgr2=0x7f1bf40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.328+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/3692103765 shutdown_connections 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.328+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/3692103765 wait complete. 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.329+0000 7f1bf8d9d640 10 monclient: build_initial_monmap 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.329+0000 7f1bf8d9d640 10 monclient: monmap: 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:37:40.329 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.329+0000 7f1bf8d9d640 1 librados: starting msgr at 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.329+0000 7f1bf8d9d640 1 librados: starting objecter 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.329+0000 7f1bf8d9d640 1 Processor -- start 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.329+0000 7f1bf8d9d640 1 -- start start 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.329+0000 7f1bf8d9d640 1 librados: setting wanted keys 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.329+0000 7f1bf8d9d640 1 librados: calling monclient init 2024-03-25T19:37:40.330 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.329+0000 7f1bf8d9d640 10 monclient: init 2024-03-25T19:37:40.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.330+0000 7f1bf8d9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:40.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.330+0000 7f1bf8d9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:40.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.330+0000 7f1bf8d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf40a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:40.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.330+0000 7f1bf8d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:40.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.330+0000 7f1bf8d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf41430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:40.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.330+0000 7f1bf8d9d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf40abae0 con 0x7f1bf41430c0 2024-03-25T19:37:40.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.330+0000 7f1bf8d9d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf40acb60 con 0x7f1bf413f710 2024-03-25T19:37:40.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.330+0000 7f1bf8d9d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf40abd50 con 0x7f1bf40a98f0 2024-03-25T19:37:40.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.330+0000 7f1bf8d9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:40.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.330+0000 7f1bf8d9d640 10 monclient(hunting): authenticate will time out at 4652.671232s 2024-03-25T19:37:40.341 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.342+0000 7f1bf259c640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1bf40a98f0 0x7f1bf4138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36254/0 (socket says 172.21.15.107:36254) 2024-03-25T19:37:40.341 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.342+0000 7f1bf259c640 1 -- 172.21.15.107:0/1280416543 learned_addr learned my addr 172.21.15.107:0/1280416543 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:40.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.345+0000 7f1be2ffd640 10 client.?.objecter ms_handle_connect 0x7f1bf40a98f0 2024-03-25T19:37:40.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.345+0000 7f1be2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:40.352 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.353+0000 7f1be2ffd640 10 client.?.objecter ms_handle_connect 0x7f1bf41430c0 2024-03-25T19:37:40.352 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.353+0000 7f1be2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:40.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.358+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115871582 0 0) 0x7f1bf40abd50 con 0x7f1bf40a98f0 2024-03-25T19:37:40.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.358+0000 7f1be2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:40.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.358+0000 7f1be2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:40.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.358+0000 7f1be2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:40.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.358+0000 7f1be2ffd640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:37:40.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.358+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd4004eb0 con 0x7f1bf40a98f0 2024-03-25T19:37:40.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.360+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668233887 0 0) 0x7f1bf40abae0 con 0x7f1bf41430c0 2024-03-25T19:37:40.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.360+0000 7f1be2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:40.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.360+0000 7f1be2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:40.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.360+0000 7f1be2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:40.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.360+0000 7f1be2ffd640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:37:40.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.360+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf40abd50 con 0x7f1bf41430c0 2024-03-25T19:37:40.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.360+0000 7f1be2ffd640 10 client.?.objecter ms_handle_connect 0x7f1bf413f710 2024-03-25T19:37:40.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.360+0000 7f1be2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:40.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3324096641 0 0) 0x7f1bf40abd50 con 0x7f1bf41430c0 2024-03-25T19:37:40.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bf40abae0 con 0x7f1bf41430c0 2024-03-25T19:37:40.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1bec001b50 con 0x7f1bf41430c0 2024-03-25T19:37:40.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:40.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:40.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:40.371 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:40.371 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676026554 0 0) 0x7f1bf40acb60 con 0x7f1bf413f710 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:37:40.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.371+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf40abd50 con 0x7f1bf413f710 2024-03-25T19:37:40.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.378+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 403516746 0 0) 0x7f1bf40abae0 con 0x7f1bf41430c0 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.378+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 >> v1:172.21.15.107:6790/0 conn(0x7f1bf40a98f0 legacy=0x7f1bf4138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.378+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 >> v1:172.21.15.195:6789/0 conn(0x7f1bf413f710 legacy=0x7f1bf40ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.378+0000 7f1be2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.378+0000 7f1be2ffd640 1 monclient: found mon.a 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.378+0000 7f1be2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.378+0000 7f1be2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.378+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf40ac030 con 0x7f1bf41430c0 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.378+0000 7f1be2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.378+0000 7f1be2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.378+0000 7f1be2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.379+0000 7f1bf8d9d640 5 monclient: authenticate success, global_id 4302 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.379+0000 7f1bf8d9d640 10 monclient: _renew_subs 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.379+0000 7f1bf8d9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.379+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/1280416543 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf41390d0 con 0x7f1bf41430c0 2024-03-25T19:37:40.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.379+0000 7f1bf8d9d640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:40.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.379+0000 7f1bf8d9d640 10 monclient: _renew_subs 2024-03-25T19:37:40.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.379+0000 7f1bf8d9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:40.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.379+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/1280416543 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf4136520 con 0x7f1bf41430c0 2024-03-25T19:37:40.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.379+0000 7f1bf8d9d640 1 librados: init done 2024-03-25T19:37:40.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.380+0000 7f1bf8d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:40.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.380+0000 7f1bf8d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:40.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.380+0000 7f1bf8d9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:40.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.380+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/1280416543 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf40627b0 con 0x7f1bf41430c0 2024-03-25T19:37:40.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.686+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bec002040 con 0x7f1bf41430c0 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.686+0000 7f1be2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.686+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1bec003b10 con 0x7f1bf41430c0 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.686+0000 7f1be2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.686+0000 7f1be2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.686+0000 7f1be2ffd640 10 monclient: dump: 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:40.686 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:40.687 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:40.687 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:40.687 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:40.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.686+0000 7f1bf0d99640 4 set_mon_vals no callback set 2024-03-25T19:37:40.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.687+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f1bec003d30 con 0x7f1bf41430c0 2024-03-25T19:37:40.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.687+0000 7f1bf259c640 10 monclient: get_auth_request con 0x7f1bd406ddc0 auth_method 0 2024-03-25T19:37:40.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.687+0000 7f1be2ffd640 10 client.4302.objecter ms_handle_connect 0x7f1bd406ddc0 2024-03-25T19:37:40.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.692+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f1bec053870 con 0x7f1bf41430c0 2024-03-25T19:37:40.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.692+0000 7f1be2ffd640 10 client.4302.objecter ms_dispatch 0x7f1bf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:40.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.692+0000 7f1be2ffd640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:40.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.692+0000 7f1be2ffd640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:40.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.692+0000 7f1be2ffd640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-25T19:37:40.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.692+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bec04f1b0 con 0x7f1bf41430c0 2024-03-25T19:37:40.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.692+0000 7f1be2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:40.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.692+0000 7f1be2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:40.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.947+0000 7f1bf8d9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:40.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.947+0000 7f1bf8d9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:40.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.947+0000 7f1bf8d9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:40.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.947+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/1280416543 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bf413ce70 con 0x7f1bf41430c0 2024-03-25T19:37:40.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.970+0000 7f1be2ffd640 1 -- 172.21.15.107:0/1280416543 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f1bec0512b0 con 0x7f1bf41430c0 2024-03-25T19:37:40.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.970+0000 7f1be2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:40.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.970+0000 7f1be2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:40.970 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:37:40.970 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:40.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 10 librados: watch_flush enter 2024-03-25T19:37:40.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 10 librados: watch_flush exit 2024-03-25T19:37:40.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-25T19:37:40.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 10 client.4302.objecter successfully canceled tick 2024-03-25T19:37:40.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/1280416543 >> v1:172.21.15.195:6800/2984408421 conn(0x7f1bd406ddc0 legacy=0x7f1bd4070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:40.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 10 monclient: shutdown 2024-03-25T19:37:40.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:40.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/1280416543 >> v1:172.21.15.107:6789/0 conn(0x7f1bf41430c0 legacy=0x7f1bf41e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:40.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/1280416543 shutdown_connections 2024-03-25T19:37:40.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/1280416543 >> 172.21.15.107:0/1280416543 conn(0x7f1bf40b7ea0 msgr2=0x7f1bf4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:40.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/1280416543 shutdown_connections 2024-03-25T19:37:40.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 1 -- 172.21.15.107:0/1280416543 wait complete. 2024-03-25T19:37:40.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:40.973+0000 7f1bf8d9d640 1 librados: shutdown 2024-03-25T19:37:40.983 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:44.555 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:02:14.871 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.565 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:02:14.881 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.606 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:02:14.922 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.623 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:02:14.940 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.626 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:02:14.942 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.632 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:02:14.919 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.636 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:02:14.923 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.658 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:02:14.945 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.665 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:02:14.981 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:47.789 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:47.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.905+0000 7f5f2436c640 10 monclient: get_monmap_and_config 2024-03-25T19:37:47.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.905+0000 7f5f2436c640 10 monclient: build_initial_monmap 2024-03-25T19:37:47.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.905+0000 7f5f2436c640 10 monclient: monmap: 2024-03-25T19:37:47.904 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:47.904 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:47.904 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:37:47.906145+0000 2024-03-25T19:37:47.905 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:37:47.906145+0000 2024-03-25T19:37:47.905 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:47.905 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:47.905 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:47.905 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:37:47.905 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:37:47.905 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:47.905 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.905+0000 7f5f2436c640 1 Processor -- start 2024-03-25T19:37:47.905 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.905+0000 7f5f2436c640 1 -- start start 2024-03-25T19:37:47.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.905+0000 7f5f2436c640 10 monclient: init 2024-03-25T19:37:47.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.906+0000 7f5f2436c640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:47.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.906+0000 7f5f2436c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:47.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.906+0000 7f5f2436c640 10 monclient(hunting): picked mon.noname-b con 0x7f5f1c0b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:47.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.906+0000 7f5f2436c640 10 monclient(hunting): picked mon.noname-a con 0x7f5f1c0b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:47.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.906+0000 7f5f2436c640 10 monclient(hunting): picked mon.noname-c con 0x7f5f1c14be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:47.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.906+0000 7f5f2436c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c0af940 con 0x7f5f1c0b1730 2024-03-25T19:37:47.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.906+0000 7f5f2436c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c0b0a70 con 0x7f5f1c14be90 2024-03-25T19:37:47.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.906+0000 7f5f2436c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c0b0d50 con 0x7f5f1c0b4d60 2024-03-25T19:37:47.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.906+0000 7f5f2436c640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:47.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.906+0000 7f5f2436c640 10 monclient(hunting): authenticate will time out at 4660.247671s 2024-03-25T19:37:47.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.912+0000 7f5f22108640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f5f1c0b4d60 0x7f5f1c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43894/0 (socket says 172.21.15.107:43894) 2024-03-25T19:37:47.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.912+0000 7f5f22108640 1 -- 172.21.15.107:0/2009764642 learned_addr learned my addr 172.21.15.107:0/2009764642 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.924+0000 7f5f21106640 1 -- 172.21.15.107:0/2009764642 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480336477 0 0) 0x7f5f1c0b0d50 con 0x7f5f1c0b4d60 2024-03-25T19:37:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.924+0000 7f5f21106640 10 monclient(hunting): _negotiate 2024-03-25T19:37:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.924+0000 7f5f21106640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.924+0000 7f5f21106640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.924+0000 7f5f21106640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:37:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.924+0000 7f5f21106640 1 -- 172.21.15.107:0/2009764642 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f08003520 con 0x7f5f1c0b4d60 2024-03-25T19:37:47.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.927+0000 7f5f21106640 1 -- 172.21.15.107:0/2009764642 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867733585 0 0) 0x7f5f1c0af940 con 0x7f5f1c0b1730 2024-03-25T19:37:47.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.927+0000 7f5f21106640 10 monclient(hunting): _negotiate 2024-03-25T19:37:47.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.927+0000 7f5f21106640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:47.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.927+0000 7f5f21106640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:47.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.927+0000 7f5f21106640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:37:47.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.927+0000 7f5f21106640 1 -- 172.21.15.107:0/2009764642 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f1c0b0d50 con 0x7f5f1c0b1730 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 1 -- 172.21.15.107:0/2009764642 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 112633001 0 0) 0x7f5f1c0b0d50 con 0x7f5f1c0b1730 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 1 -- 172.21.15.107:0/2009764642 >> v1:172.21.15.107:6790/0 conn(0x7f5f1c0b4d60 legacy=0x7f5f1c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 1 -- 172.21.15.107:0/2009764642 >> v1:172.21.15.195:6789/0 conn(0x7f5f1c14be90 legacy=0x7f5f1c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 10 monclient: _finish_hunting 0 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 1 monclient: found mon.noname-a 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 1 -- 172.21.15.107:0/2009764642 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c0b1030 con 0x7f5f1c0b1730 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 10 monclient: _finish_auth 0 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 10 monclient: _check_auth_tickets 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 1 -- 172.21.15.107:0/2009764642 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5f0c001670 con 0x7f5f1c0b1730 2024-03-25T19:37:47.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f2436c640 5 monclient: authenticate success, global_id 4305 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f2436c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f21106640 10 monclient: dump: 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:47.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.935+0000 7f5f2436c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:47.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.943+0000 7f5f21106640 1 -- 172.21.15.107:0/2009764642 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f0c0017d0 con 0x7f5f1c0b1730 2024-03-25T19:37:47.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f21106640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:47.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f21106640 1 -- 172.21.15.107:0/2009764642 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5f0c001ba0 con 0x7f5f1c0b1730 2024-03-25T19:37:47.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f21106640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:47.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f21106640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:47.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f21106640 10 monclient: dump: 2024-03-25T19:37:47.943 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:47.944 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:47.944 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:47.944 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:47.944 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:47.944 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:47.944 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:47.944 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:47.944 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:47.944 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:47.944 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f2436c640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:47.944 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f2436c640 4 set_mon_vals no callback set 2024-03-25T19:37:47.945 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f2436c640 10 monclient: shutdown 2024-03-25T19:37:47.945 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f2436c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:47.945 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f2436c640 1 -- 172.21.15.107:0/2009764642 >> v1:172.21.15.107:6789/0 conn(0x7f5f1c0b1730 legacy=0x7f5f1c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.945 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f2436c640 1 -- 172.21.15.107:0/2009764642 shutdown_connections 2024-03-25T19:37:47.945 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f2436c640 1 -- 172.21.15.107:0/2009764642 >> 172.21.15.107:0/2009764642 conn(0x7f5f1c0617a0 msgr2=0x7f5f1c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f2436c640 1 -- 172.21.15.107:0/2009764642 shutdown_connections 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.944+0000 7f5f2436c640 1 -- 172.21.15.107:0/2009764642 wait complete. 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.945+0000 7f5f2436c640 10 monclient: build_initial_monmap 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.945+0000 7f5f2436c640 10 monclient: monmap: 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.945+0000 7f5f2436c640 1 librados: starting msgr at 2024-03-25T19:37:47.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.945+0000 7f5f2436c640 1 librados: starting objecter 2024-03-25T19:37:47.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.945+0000 7f5f2436c640 1 Processor -- start 2024-03-25T19:37:47.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 1 -- start start 2024-03-25T19:37:47.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 1 librados: setting wanted keys 2024-03-25T19:37:47.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 1 librados: calling monclient init 2024-03-25T19:37:47.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 10 monclient: init 2024-03-25T19:37:47.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:47.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:47.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 10 monclient(hunting): picked mon.noname-b con 0x7f5f1c0b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:47.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 10 monclient(hunting): picked mon.noname-a con 0x7f5f1c0b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:47.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 10 monclient(hunting): picked mon.noname-c con 0x7f5f1c14be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:47.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c1327e0 con 0x7f5f1c0b4d60 2024-03-25T19:37:47.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c137ee0 con 0x7f5f1c0b1730 2024-03-25T19:37:47.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c138070 con 0x7f5f1c14be90 2024-03-25T19:37:47.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:47.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.946+0000 7f5f2436c640 10 monclient(hunting): authenticate will time out at 4660.287658s 2024-03-25T19:37:47.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.953+0000 7f5f21907640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5f1c0b4d60 0x7f5f1c1320d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34744/0 (socket says 172.21.15.107:34744) 2024-03-25T19:37:47.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.953+0000 7f5f21907640 1 -- 172.21.15.107:0/592965550 learned_addr learned my addr 172.21.15.107:0/592965550 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:47.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.955+0000 7f5f12ffd640 10 client.?.objecter ms_handle_connect 0x7f5f1c14be90 2024-03-25T19:37:47.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.956+0000 7f5f12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:47.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.958+0000 7f5f12ffd640 10 client.?.objecter ms_handle_connect 0x7f5f1c0b4d60 2024-03-25T19:37:47.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.958+0000 7f5f12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:47.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.960+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416283529 0 0) 0x7f5f1c138070 con 0x7f5f1c14be90 2024-03-25T19:37:47.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.960+0000 7f5f12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:47.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.960+0000 7f5f12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:47.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.960+0000 7f5f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:47.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.960+0000 7f5f12ffd640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:37:47.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.960+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f00004eb0 con 0x7f5f1c14be90 2024-03-25T19:37:47.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.966+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666692387 0 0) 0x7f5f1c1327e0 con 0x7f5f1c0b4d60 2024-03-25T19:37:47.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.966+0000 7f5f12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:47.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.966+0000 7f5f12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:47.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.966+0000 7f5f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:47.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.966+0000 7f5f12ffd640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:37:47.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.966+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f1c138070 con 0x7f5f1c0b4d60 2024-03-25T19:37:47.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.974+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3605451887 0 0) 0x7f5f00004eb0 con 0x7f5f1c14be90 2024-03-25T19:37:47.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.974+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f1c1327e0 con 0x7f5f1c14be90 2024-03-25T19:37:47.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.974+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5f14001b90 con 0x7f5f1c14be90 2024-03-25T19:37:47.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.974+0000 7f5f12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:47.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.974+0000 7f5f12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:47.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.974+0000 7f5f12ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:47.974 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:47.975 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:47.975 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:47.975 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:47.975 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:47.975 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:47.975 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:47.975 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:47.975 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:47.975 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:47.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.974+0000 7f5f12ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:47.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.977+0000 7f5f12ffd640 10 client.?.objecter ms_handle_connect 0x7f5f1c0b1730 2024-03-25T19:37:47.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.977+0000 7f5f12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:47.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.987+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353746599 0 0) 0x7f5f1c137ee0 con 0x7f5f1c0b1730 2024-03-25T19:37:47.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.987+0000 7f5f12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:47.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.987+0000 7f5f12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:47.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.987+0000 7f5f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:47.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.987+0000 7f5f12ffd640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:37:47.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.987+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f00004eb0 con 0x7f5f1c0b1730 2024-03-25T19:37:47.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 480249386 0 0) 0x7f5f1c1327e0 con 0x7f5f1c14be90 2024-03-25T19:37:47.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 >> v1:172.21.15.195:6789/0 conn(0x7f5f1c0b1730 legacy=0x7f5f1c138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 >> v1:172.21.15.107:6789/0 conn(0x7f5f1c0b4d60 legacy=0x7f5f1c1320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:47.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f12ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:47.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f12ffd640 1 monclient: found mon.c 2024-03-25T19:37:47.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:47.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f12ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c144b90 con 0x7f5f1c14be90 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f12ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f12ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f2436c640 5 monclient: authenticate success, global_id 4313 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f2436c640 10 monclient: _renew_subs 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f2436c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f2436c640 1 -- 172.21.15.107:0/592965550 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1c144d80 con 0x7f5f1c14be90 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f2436c640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f2436c640 10 monclient: _renew_subs 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f2436c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f2436c640 1 -- 172.21.15.107:0/592965550 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1c13b870 con 0x7f5f1c14be90 2024-03-25T19:37:47.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.992+0000 7f5f2436c640 1 librados: init done 2024-03-25T19:37:47.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.993+0000 7f5f2436c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:47.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.993+0000 7f5f2436c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:47.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.993+0000 7f5f2436c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:47.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:47.993+0000 7f5f2436c640 1 -- 172.21.15.107:0/592965550 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1c0b6fb0 con 0x7f5f1c14be90 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.243+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f14002080 con 0x7f5f1c14be90 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.243+0000 7f5f12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.243+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5f14003b30 con 0x7f5f1c14be90 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.243+0000 7f5f12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.243+0000 7f5f12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.243+0000 7f5f12ffd640 10 monclient: dump: 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:48.243 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:48.244 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:48.244 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:48.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.243+0000 7f5f20905640 4 set_mon_vals no callback set 2024-03-25T19:37:48.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.244+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f5f14003d50 con 0x7f5f1c14be90 2024-03-25T19:37:48.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.244+0000 7f5f22108640 10 monclient: get_auth_request con 0x7f5f0006dda0 auth_method 0 2024-03-25T19:37:48.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.245+0000 7f5f12ffd640 10 client.4313.objecter ms_handle_connect 0x7f5f0006dda0 2024-03-25T19:37:48.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.248+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f5f14053830 con 0x7f5f1c14be90 2024-03-25T19:37:48.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.248+0000 7f5f12ffd640 10 client.4313.objecter ms_dispatch 0x7f5f1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:48.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.248+0000 7f5f12ffd640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:48.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.248+0000 7f5f12ffd640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:48.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.248+0000 7f5f12ffd640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-25T19:37:48.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.248+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f1404f170 con 0x7f5f1c14be90 2024-03-25T19:37:48.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.248+0000 7f5f12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:48.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.248+0000 7f5f12ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:48.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.468+0000 7f5f2436c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:48.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.468+0000 7f5f2436c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:48.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.468+0000 7f5f2436c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:37:48.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.468+0000 7f5f2436c640 1 -- 172.21.15.107:0/592965550 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f1c139ae0 con 0x7f5f1c14be90 2024-03-25T19:37:48.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.492+0000 7f5f12ffd640 1 -- 172.21.15.107:0/592965550 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f5f14051270 con 0x7f5f1c14be90 2024-03-25T19:37:48.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.492+0000 7f5f12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:48.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.492+0000 7f5f12ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:48.492 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:37:48.492 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 10 librados: watch_flush enter 2024-03-25T19:37:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 10 librados: watch_flush exit 2024-03-25T19:37:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-25T19:37:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 10 client.4313.objecter successfully canceled tick 2024-03-25T19:37:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 1 -- 172.21.15.107:0/592965550 >> v1:172.21.15.195:6800/2984408421 conn(0x7f5f0006dda0 legacy=0x7f5f00070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 10 monclient: shutdown 2024-03-25T19:37:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 1 -- 172.21.15.107:0/592965550 >> v1:172.21.15.107:6790/0 conn(0x7f5f1c14be90 legacy=0x7f5f1c13af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 1 -- 172.21.15.107:0/592965550 shutdown_connections 2024-03-25T19:37:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 1 -- 172.21.15.107:0/592965550 >> 172.21.15.107:0/592965550 conn(0x7f5f1c0617a0 msgr2=0x7f5f1c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:48.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 1 -- 172.21.15.107:0/592965550 shutdown_connections 2024-03-25T19:37:48.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 1 -- 172.21.15.107:0/592965550 wait complete. 2024-03-25T19:37:48.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:48.495+0000 7f5f2436c640 1 librados: shutdown 2024-03-25T19:37:48.507 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:55.314 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:55.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.429+0000 7ff88960f640 10 monclient: get_monmap_and_config 2024-03-25T19:37:55.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.429+0000 7ff88960f640 10 monclient: build_initial_monmap 2024-03-25T19:37:55.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.430+0000 7ff88960f640 10 monclient: monmap: 2024-03-25T19:37:55.429 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:55.429 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:55.429 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:37:55.430928+0000 2024-03-25T19:37:55.430 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:37:55.430928+0000 2024-03-25T19:37:55.430 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:55.430 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:55.430 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:55.430 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:37:55.430 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:37:55.430 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:55.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.430+0000 7ff88960f640 1 Processor -- start 2024-03-25T19:37:55.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 1 -- start start 2024-03-25T19:37:55.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 10 monclient: init 2024-03-25T19:37:55.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:55.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:55.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 10 monclient(hunting): picked mon.noname-b con 0x7ff8840a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:55.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 10 monclient(hunting): picked mon.noname-c con 0x7ff88413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:55.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 10 monclient(hunting): picked mon.noname-a con 0x7ff8841430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:55.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff884142520 con 0x7ff8841430c0 2024-03-25T19:37:55.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840b28a0 con 0x7ff88413f710 2024-03-25T19:37:55.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840b2a30 con 0x7ff8840a98f0 2024-03-25T19:37:55.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:55.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.431+0000 7ff88960f640 10 monclient(hunting): authenticate will time out at 4667.772478s 2024-03-25T19:37:55.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.439+0000 7ff87bfff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff88413f710 0x7ff8841429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:60866/0 (socket says 172.21.15.107:60866) 2024-03-25T19:37:55.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.439+0000 7ff87bfff640 1 -- 172.21.15.107:0/2693869483 learned_addr learned my addr 172.21.15.107:0/2693869483 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:55.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.453+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936875517 0 0) 0x7ff8840b28a0 con 0x7ff88413f710 2024-03-25T19:37:55.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.453+0000 7ff8827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:55.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.453+0000 7ff8827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:55.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.453+0000 7ff8827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:55.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.453+0000 7ff8827fc640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:37:55.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.453+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff870003520 con 0x7ff88413f710 2024-03-25T19:37:55.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.457+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389503889 0 0) 0x7ff884142520 con 0x7ff8841430c0 2024-03-25T19:37:55.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.457+0000 7ff8827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:55.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.457+0000 7ff8827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:55.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.457+0000 7ff8827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:55.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.457+0000 7ff8827fc640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:37:55.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.457+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8840b28a0 con 0x7ff8841430c0 2024-03-25T19:37:55.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.465+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396162416 0 0) 0x7ff8840b2a30 con 0x7ff8840a98f0 2024-03-25T19:37:55.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.465+0000 7ff8827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:55.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.465+0000 7ff8827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:55.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.465+0000 7ff8827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:55.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.465+0000 7ff8827fc640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:37:55.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.465+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff884142520 con 0x7ff8840a98f0 2024-03-25T19:37:55.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1149646684 0 0) 0x7ff870003520 con 0x7ff88413f710 2024-03-25T19:37:55.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 >> v1:172.21.15.107:6790/0 conn(0x7ff8840a98f0 legacy=0x7ff8840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:55.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 >> v1:172.21.15.107:6789/0 conn(0x7ff8841430c0 legacy=0x7ff88414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:55.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:55.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 1 monclient: found mon.noname-c 2024-03-25T19:37:55.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:55.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:37:55.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8840b44c0 con 0x7ff88413f710 2024-03-25T19:37:55.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:55.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:55.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff86c001690 con 0x7ff88413f710 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff8827fc640 10 monclient: dump: 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff88960f640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:37:55.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.467+0000 7ff88960f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.470+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff86c0017f0 con 0x7ff88413f710 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.470+0000 7ff8827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.470+0000 7ff8827fc640 1 -- 172.21.15.107:0/2693869483 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff86c001c00 con 0x7ff88413f710 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.470+0000 7ff88960f640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.470+0000 7ff88960f640 4 set_mon_vals no callback set 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.470+0000 7ff8827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.470+0000 7ff8827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.470+0000 7ff8827fc640 10 monclient: dump: 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:55.470 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:55.471 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:55.471 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:55.471 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:55.471 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:55.471 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.471+0000 7ff88960f640 10 monclient: shutdown 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.471+0000 7ff88960f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.471+0000 7ff88960f640 1 -- 172.21.15.107:0/2693869483 >> v1:172.21.15.195:6789/0 conn(0x7ff88413f710 legacy=0x7ff8841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.471+0000 7ff88960f640 1 -- 172.21.15.107:0/2693869483 shutdown_connections 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.471+0000 7ff88960f640 1 -- 172.21.15.107:0/2693869483 >> 172.21.15.107:0/2693869483 conn(0x7ff8840b7ea0 msgr2=0x7ff8840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.471+0000 7ff88960f640 1 -- 172.21.15.107:0/2693869483 shutdown_connections 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.471+0000 7ff88960f640 1 -- 172.21.15.107:0/2693869483 wait complete. 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.472+0000 7ff88960f640 10 monclient: build_initial_monmap 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.472+0000 7ff88960f640 10 monclient: monmap: 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:55.473 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.472+0000 7ff88960f640 1 librados: starting msgr at 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.472+0000 7ff88960f640 1 librados: starting objecter 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.472+0000 7ff88960f640 1 Processor -- start 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.472+0000 7ff88960f640 1 -- start start 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.472+0000 7ff88960f640 1 librados: setting wanted keys 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.472+0000 7ff88960f640 1 librados: calling monclient init 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.472+0000 7ff88960f640 10 monclient: init 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.473+0000 7ff88960f640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:55.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.473+0000 7ff88960f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:55.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.473+0000 7ff88960f640 10 monclient(hunting): picked mon.noname-b con 0x7ff8840a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:37:55.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.473+0000 7ff88960f640 10 monclient(hunting): picked mon.noname-c con 0x7ff88413f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:37:55.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.473+0000 7ff88960f640 10 monclient(hunting): picked mon.noname-a con 0x7ff8841430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:37:55.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.473+0000 7ff88960f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8841d77a0 con 0x7ff8841430c0 2024-03-25T19:37:55.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.473+0000 7ff88960f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff884138e40 con 0x7ff8840a98f0 2024-03-25T19:37:55.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.473+0000 7ff88960f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840aba30 con 0x7ff88413f710 2024-03-25T19:37:55.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.473+0000 7ff88960f640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:55.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.473+0000 7ff88960f640 10 monclient(hunting): authenticate will time out at 4667.813960s 2024-03-25T19:37:55.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.476+0000 7ff8837fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff8841430c0 0x7ff884134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36498/0 (socket says 172.21.15.107:36498) 2024-03-25T19:37:55.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.476+0000 7ff8837fe640 1 -- 172.21.15.107:0/1454894185 learned_addr learned my addr 172.21.15.107:0/1454894185 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:37:55.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.476+0000 7ff87b7fe640 10 client.?.objecter ms_handle_connect 0x7ff8841430c0 2024-03-25T19:37:55.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.476+0000 7ff87b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:55.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.486+0000 7ff87b7fe640 10 client.?.objecter ms_handle_connect 0x7ff88413f710 2024-03-25T19:37:55.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.486+0000 7ff87b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:55.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.487+0000 7ff87b7fe640 10 client.?.objecter ms_handle_connect 0x7ff8840a98f0 2024-03-25T19:37:55.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.487+0000 7ff87b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:55.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.490+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256330080 0 0) 0x7ff8841d77a0 con 0x7ff8841430c0 2024-03-25T19:37:55.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.490+0000 7ff87b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:55.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.490+0000 7ff87b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:55.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.490+0000 7ff87b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:55.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.490+0000 7ff87b7fe640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:37:55.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.490+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff864004eb0 con 0x7ff8841430c0 2024-03-25T19:37:55.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.501+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1337537507 0 0) 0x7ff864004eb0 con 0x7ff8841430c0 2024-03-25T19:37:55.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.501+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8841d77a0 con 0x7ff8841430c0 2024-03-25T19:37:55.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.501+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949720011 0 0) 0x7ff884138e40 con 0x7ff8840a98f0 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.501+0000 7ff87b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.501+0000 7ff87b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.501+0000 7ff87b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.501+0000 7ff87b7fe640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.501+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff864004eb0 con 0x7ff8840a98f0 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.501+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308986297 0 0) 0x7ff8840aba30 con 0x7ff88413f710 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.502+0000 7ff87b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.502+0000 7ff87b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.502+0000 7ff87b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.502+0000 7ff87b7fe640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.502+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff884138e40 con 0x7ff88413f710 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.502+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff87c001d90 con 0x7ff8841430c0 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.502+0000 7ff87b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.502+0000 7ff87b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:55.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.502+0000 7ff87b7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:55.503 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:55.503 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:55.503 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:55.503 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:55.503 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:55.503 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:55.503 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:55.503 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:55.503 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:55.503 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:55.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.502+0000 7ff87b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:55.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.510+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2262399416 0 0) 0x7ff864004eb0 con 0x7ff8840a98f0 2024-03-25T19:37:55.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.510+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8840aba30 con 0x7ff8840a98f0 2024-03-25T19:37:55.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.510+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff874001830 con 0x7ff8840a98f0 2024-03-25T19:37:55.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.510+0000 7ff87b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:55.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.510+0000 7ff87b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:55.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.510+0000 7ff87b7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:55.511 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:55.511 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:55.511 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:55.511 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:55.511 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:55.511 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:55.511 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:55.511 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:55.511 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:55.511 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:55.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1357422208 0 0) 0x7ff8841d77a0 con 0x7ff8841430c0 2024-03-25T19:37:55.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 >> v1:172.21.15.107:6790/0 conn(0x7ff88413f710 legacy=0x7ff8841d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 >> v1:172.21.15.195:6789/0 conn(0x7ff8840a98f0 legacy=0x7ff884138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff87b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff87b7fe640 1 monclient: found mon.a 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff87b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff87b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8841d8340 con 0x7ff8841430c0 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff87b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff87b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff87b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff88960f640 5 monclient: authenticate success, global_id 4314 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff88960f640 10 monclient: _renew_subs 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff88960f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff88960f640 1 -- 172.21.15.107:0/1454894185 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff884134820 con 0x7ff8841430c0 2024-03-25T19:37:55.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff88960f640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:55.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff88960f640 10 monclient: _renew_subs 2024-03-25T19:37:55.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.515+0000 7ff88960f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:55.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.516+0000 7ff88960f640 1 -- 172.21.15.107:0/1454894185 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8841380c0 con 0x7ff8841430c0 2024-03-25T19:37:55.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.516+0000 7ff88960f640 1 librados: init done 2024-03-25T19:37:55.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.516+0000 7ff88960f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:55.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.516+0000 7ff88960f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:55.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.516+0000 7ff88960f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:55.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.516+0000 7ff88960f640 1 -- 172.21.15.107:0/1454894185 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8840627b0 con 0x7ff8841430c0 2024-03-25T19:37:55.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.822+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff87c003970 con 0x7ff8841430c0 2024-03-25T19:37:55.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.822+0000 7ff87b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:55.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.822+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff87c003d80 con 0x7ff8841430c0 2024-03-25T19:37:55.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.822+0000 7ff87b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:55.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.822+0000 7ff87b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:55.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.822+0000 7ff87b7fe640 10 monclient: dump: 2024-03-25T19:37:55.822 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:37:55.822 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:37:55.822 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:55.823 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:37:55.823 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:55.823 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:37:55.823 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:37:55.823 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:37:55.823 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:37:55.823 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:37:55.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.822+0000 7ff881ffb640 4 set_mon_vals no callback set 2024-03-25T19:37:55.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.823+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7ff87c003fa0 con 0x7ff8841430c0 2024-03-25T19:37:55.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.824+0000 7ff882ffd640 10 monclient: get_auth_request con 0x7ff86406df90 auth_method 0 2024-03-25T19:37:55.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.824+0000 7ff87b7fe640 10 client.4314.objecter ms_handle_connect 0x7ff86406df90 2024-03-25T19:37:55.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.829+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7ff87c054ae0 con 0x7ff8841430c0 2024-03-25T19:37:55.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.829+0000 7ff87b7fe640 10 client.4314.objecter ms_dispatch 0x7ff884000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:55.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.829+0000 7ff87b7fe640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:55.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.829+0000 7ff87b7fe640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:55.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.829+0000 7ff87b7fe640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-25T19:37:55.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.829+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff87c0505f0 con 0x7ff8841430c0 2024-03-25T19:37:55.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.829+0000 7ff87b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:55.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:55.829+0000 7ff87b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:56.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.088+0000 7ff88960f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:56.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.088+0000 7ff88960f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:56.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.088+0000 7ff88960f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:37:56.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.089+0000 7ff88960f640 1 -- 172.21.15.107:0/1454894185 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff88414e880 con 0x7ff8841430c0 2024-03-25T19:37:56.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.118+0000 7ff87b7fe640 1 -- 172.21.15.107:0/1454894185 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7ff87c052520 con 0x7ff8841430c0 2024-03-25T19:37:56.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.118+0000 7ff87b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:56.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.118+0000 7ff87b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:56.118 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:37:56.118 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.120+0000 7ff88960f640 10 librados: watch_flush enter 2024-03-25T19:37:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.120+0000 7ff88960f640 10 librados: watch_flush exit 2024-03-25T19:37:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.120+0000 7ff88960f640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-25T19:37:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.120+0000 7ff88960f640 10 client.4314.objecter successfully canceled tick 2024-03-25T19:37:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.121+0000 7ff88960f640 1 -- 172.21.15.107:0/1454894185 >> v1:172.21.15.195:6800/2984408421 conn(0x7ff86406df90 legacy=0x7ff864070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.121+0000 7ff88960f640 10 monclient: shutdown 2024-03-25T19:37:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.121+0000 7ff88960f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.121+0000 7ff88960f640 1 -- 172.21.15.107:0/1454894185 >> v1:172.21.15.107:6789/0 conn(0x7ff8841430c0 legacy=0x7ff884134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.121+0000 7ff88960f640 1 -- 172.21.15.107:0/1454894185 shutdown_connections 2024-03-25T19:37:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.121+0000 7ff88960f640 1 -- 172.21.15.107:0/1454894185 >> 172.21.15.107:0/1454894185 conn(0x7ff8840b7ea0 msgr2=0x7ff884074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.121+0000 7ff88960f640 1 -- 172.21.15.107:0/1454894185 shutdown_connections 2024-03-25T19:37:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.121+0000 7ff88960f640 1 -- 172.21.15.107:0/1454894185 wait complete. 2024-03-25T19:37:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:37:56.121+0000 7ff88960f640 1 librados: shutdown 2024-03-25T19:37:56.132 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:02.938 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:03.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.057+0000 7f72a9a14640 10 monclient: get_monmap_and_config 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.057+0000 7f72a9a14640 10 monclient: build_initial_monmap 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.057+0000 7f72a9a14640 10 monclient: monmap: 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:38:03.058222+0000 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:38:03.058222+0000 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:38:03.057 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:03.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 1 Processor -- start 2024-03-25T19:38:03.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 1 -- start start 2024-03-25T19:38:03.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 10 monclient: init 2024-03-25T19:38:03.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:03.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:03.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 10 monclient(hunting): picked mon.noname-a con 0x7f72a40a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:03.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 10 monclient(hunting): picked mon.noname-b con 0x7f72a413f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:03.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 10 monclient(hunting): picked mon.noname-c con 0x7f72a41430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:03.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a4142520 con 0x7f72a40a98f0 2024-03-25T19:38:03.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a40b28a0 con 0x7f72a41430c0 2024-03-25T19:38:03.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a40b2a30 con 0x7f72a413f710 2024-03-25T19:38:03.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:03.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.058+0000 7f72a9a14640 10 monclient(hunting): authenticate will time out at 4675.399594s 2024-03-25T19:38:03.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.074+0000 7f72a2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f72a40a98f0 0x7f72a40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44766/0 (socket says 172.21.15.107:44766) 2024-03-25T19:38:03.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.074+0000 7f72a2ffd640 1 -- 172.21.15.107:0/66113383 learned_addr learned my addr 172.21.15.107:0/66113383 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:03.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.090+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057812054 0 0) 0x7f72a4142520 con 0x7f72a40a98f0 2024-03-25T19:38:03.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.090+0000 7f72a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:03.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.090+0000 7f72a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:03.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.090+0000 7f72a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:03.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.090+0000 7f72a1ffb640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:38:03.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.090+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f728c003520 con 0x7f72a40a98f0 2024-03-25T19:38:03.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.094+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818655912 0 0) 0x7f72a40b2a30 con 0x7f72a413f710 2024-03-25T19:38:03.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.094+0000 7f72a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:03.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.094+0000 7f72a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:03.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.094+0000 7f72a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:03.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.094+0000 7f72a1ffb640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:38:03.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.094+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a4142520 con 0x7f72a413f710 2024-03-25T19:38:03.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.095+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29458889 0 0) 0x7f72a40b28a0 con 0x7f72a41430c0 2024-03-25T19:38:03.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.095+0000 7f72a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:03.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.095+0000 7f72a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:03.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.095+0000 7f72a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:03.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.095+0000 7f72a1ffb640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:38:03.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.095+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a40b2a30 con 0x7f72a41430c0 2024-03-25T19:38:03.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1203783965 0 0) 0x7f728c003520 con 0x7f72a40a98f0 2024-03-25T19:38:03.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 >> v1:172.21.15.107:6790/0 conn(0x7f72a413f710 legacy=0x7f72a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:03.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 >> v1:172.21.15.195:6789/0 conn(0x7f72a41430c0 legacy=0x7f72a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:03.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:03.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 1 monclient: found mon.noname-a 2024-03-25T19:38:03.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:03.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:38:03.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a40b44c0 con 0x7f72a40a98f0 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7294001690 con 0x7f72a40a98f0 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a9a14640 5 monclient: authenticate success, global_id 4317 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a9a14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a1ffb640 10 monclient: dump: 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:03.101 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:03.102 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:03.102 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:03.102 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:03.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.100+0000 7f72a9a14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:03.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.106+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72940017f0 con 0x7f72a40a98f0 2024-03-25T19:38:03.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.106+0000 7f72a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.106+0000 7f72a1ffb640 1 -- 172.21.15.107:0/66113383 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7294001c00 con 0x7f72a40a98f0 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.106+0000 7f72a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.106+0000 7f72a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.106+0000 7f72a1ffb640 10 monclient: dump: 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:03.106 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:03.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.106+0000 7f72a9a14640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:03.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.106+0000 7f72a9a14640 4 set_mon_vals no callback set 2024-03-25T19:38:03.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.107+0000 7f72a9a14640 10 monclient: shutdown 2024-03-25T19:38:03.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.107+0000 7f72a9a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:03.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.107+0000 7f72a9a14640 1 -- 172.21.15.107:0/66113383 >> v1:172.21.15.107:6789/0 conn(0x7f72a40a98f0 legacy=0x7f72a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:03.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.107+0000 7f72a9a14640 1 -- 172.21.15.107:0/66113383 shutdown_connections 2024-03-25T19:38:03.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.107+0000 7f72a9a14640 1 -- 172.21.15.107:0/66113383 >> 172.21.15.107:0/66113383 conn(0x7f72a40b7ea0 msgr2=0x7f72a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:03.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.107+0000 7f72a9a14640 1 -- 172.21.15.107:0/66113383 shutdown_connections 2024-03-25T19:38:03.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.107+0000 7f72a9a14640 1 -- 172.21.15.107:0/66113383 wait complete. 2024-03-25T19:38:03.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.108+0000 7f72a9a14640 10 monclient: build_initial_monmap 2024-03-25T19:38:03.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.108+0000 7f72a9a14640 10 monclient: monmap: 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.108+0000 7f72a9a14640 1 librados: starting msgr at 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.108+0000 7f72a9a14640 1 librados: starting objecter 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.108+0000 7f72a9a14640 1 Processor -- start 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.108+0000 7f72a9a14640 1 -- start start 2024-03-25T19:38:03.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 1 librados: setting wanted keys 2024-03-25T19:38:03.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 1 librados: calling monclient init 2024-03-25T19:38:03.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 10 monclient: init 2024-03-25T19:38:03.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:03.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:03.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 10 monclient(hunting): picked mon.noname-c con 0x7f72a40a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:03.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 10 monclient(hunting): picked mon.noname-a con 0x7f72a413f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:03.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 10 monclient(hunting): picked mon.noname-b con 0x7f72a41430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:03.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a41369b0 con 0x7f72a413f710 2024-03-25T19:38:03.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a41366b0 con 0x7f72a41430c0 2024-03-25T19:38:03.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a4133ad0 con 0x7f72a40a98f0 2024-03-25T19:38:03.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:03.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.109+0000 7f72a9a14640 10 monclient(hunting): authenticate will time out at 4675.450550s 2024-03-25T19:38:03.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.118+0000 7f72a27fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f72a413f710 0x7f72a4132cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44824/0 (socket says 172.21.15.107:44824) 2024-03-25T19:38:03.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.118+0000 7f72a27fc640 1 -- 172.21.15.107:0/2901998170 learned_addr learned my addr 172.21.15.107:0/2901998170 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:03.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.118+0000 7f728bfff640 10 client.?.objecter ms_handle_connect 0x7f72a413f710 2024-03-25T19:38:03.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.118+0000 7f728bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:03.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.118+0000 7f728bfff640 10 client.?.objecter ms_handle_connect 0x7f72a41430c0 2024-03-25T19:38:03.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.118+0000 7f728bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:03.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.183+0000 7f728bfff640 10 client.?.objecter ms_handle_connect 0x7f72a40a98f0 2024-03-25T19:38:03.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.183+0000 7f728bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:03.200 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.201+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507459898 0 0) 0x7f72a4133ad0 con 0x7f72a40a98f0 2024-03-25T19:38:03.200 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.201+0000 7f728bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:03.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.201+0000 7f728bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:03.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.201+0000 7f728bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:03.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.201+0000 7f728bfff640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:38:03.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.201+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7290004eb0 con 0x7f72a40a98f0 2024-03-25T19:38:03.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.208+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844369518 0 0) 0x7f72a41369b0 con 0x7f72a413f710 2024-03-25T19:38:03.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.208+0000 7f728bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:03.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.208+0000 7f728bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:03.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.208+0000 7f728bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:03.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.208+0000 7f728bfff640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:38:03.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.208+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a4133ad0 con 0x7f72a413f710 2024-03-25T19:38:03.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.230+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2460592860 0 0) 0x7f7290004eb0 con 0x7f72a40a98f0 2024-03-25T19:38:03.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.230+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72a41369b0 con 0x7f72a40a98f0 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.230+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7294001b00 con 0x7f72a40a98f0 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.230+0000 7f728bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.230+0000 7f728bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.230+0000 7f728bfff640 10 monclient(hunting): dump: 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:03.230 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:03.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.230+0000 7f728bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:03.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.239+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2976984229 0 0) 0x7f72a4133ad0 con 0x7f72a413f710 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.239+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7290004eb0 con 0x7f72a413f710 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.239+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f72980019b0 con 0x7f72a413f710 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.239+0000 7f728bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.239+0000 7f728bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.239+0000 7f728bfff640 10 monclient(hunting): dump: 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:03.239 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:03.240 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:03.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168360470 0 0) 0x7f72a41366b0 con 0x7f72a41430c0 2024-03-25T19:38:03.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:03.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:03.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:03.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:38:03.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a4133ad0 con 0x7f72a41430c0 2024-03-25T19:38:03.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4177691984 0 0) 0x7f7290004eb0 con 0x7f72a413f710 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 >> v1:172.21.15.107:6790/0 conn(0x7f72a40a98f0 legacy=0x7f72a41371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 >> v1:172.21.15.195:6789/0 conn(0x7f72a41430c0 legacy=0x7f72a41333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 1 monclient: found mon.a 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.257+0000 7f728bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f728bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4136e40 con 0x7f72a413f710 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f728bfff640 10 monclient: _finish_auth 0 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f728bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f728bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f72a9a14640 5 monclient: authenticate success, global_id 4320 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f72a9a14640 10 monclient: _renew_subs 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f72a9a14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:38:03.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f72a9a14640 1 -- 172.21.15.107:0/2901998170 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a41378f0 con 0x7f72a413f710 2024-03-25T19:38:03.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f72a9a14640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:03.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f72a9a14640 10 monclient: _renew_subs 2024-03-25T19:38:03.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f72a9a14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:38:03.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f72a9a14640 1 -- 172.21.15.107:0/2901998170 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a40b1480 con 0x7f72a413f710 2024-03-25T19:38:03.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f72a9a14640 1 librados: init done 2024-03-25T19:38:03.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.258+0000 7f72a9a14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:03.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.259+0000 7f72a9a14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:03.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.259+0000 7f72a9a14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:38:03.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.259+0000 7f72a9a14640 1 -- 172.21.15.107:0/2901998170 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a4137030 con 0x7f72a413f710 2024-03-25T19:38:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.596+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7298003610 con 0x7f72a413f710 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.596+0000 7f728bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.596+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7298003a20 con 0x7f72a413f710 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.596+0000 7f728bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.596+0000 7f728bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.596+0000 7f728bfff640 10 monclient: dump: 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:03.596 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:03.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.596+0000 7f72a17fa640 4 set_mon_vals no callback set 2024-03-25T19:38:03.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.597+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f7298003c40 con 0x7f72a413f710 2024-03-25T19:38:03.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.597+0000 7f72a2ffd640 10 monclient: get_auth_request con 0x7f729006dee0 auth_method 0 2024-03-25T19:38:03.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.598+0000 7f728bfff640 10 client.4320.objecter ms_handle_connect 0x7f729006dee0 2024-03-25T19:38:03.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.604+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f7298054810 con 0x7f72a413f710 2024-03-25T19:38:03.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.604+0000 7f728bfff640 10 client.4320.objecter ms_dispatch 0x7f72a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:03.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.604+0000 7f728bfff640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:03.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.604+0000 7f728bfff640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:03.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.604+0000 7f728bfff640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-25T19:38:03.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.604+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7298050320 con 0x7f72a413f710 2024-03-25T19:38:03.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.604+0000 7f728bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:03.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.604+0000 7f728bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:03.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.851+0000 7f72a9a14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:03.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.851+0000 7f72a9a14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:03.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.851+0000 7f72a9a14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:38:03.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.851+0000 7f72a9a14640 1 -- 172.21.15.107:0/2901998170 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72a4136b60 con 0x7f72a413f710 2024-03-25T19:38:03.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.899+0000 7f728bfff640 1 -- 172.21.15.107:0/2901998170 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f72a4136b60 con 0x7f72a413f710 2024-03-25T19:38:03.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.899+0000 7f728bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:03.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.899+0000 7f728bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:03.899 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:38:03.899 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:03.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 10 librados: watch_flush enter 2024-03-25T19:38:03.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 10 librados: watch_flush exit 2024-03-25T19:38:03.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-25T19:38:03.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 10 client.4320.objecter successfully canceled tick 2024-03-25T19:38:03.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 1 -- 172.21.15.107:0/2901998170 >> v1:172.21.15.195:6800/2984408421 conn(0x7f729006dee0 legacy=0x7f72900703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:03.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 10 monclient: shutdown 2024-03-25T19:38:03.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:03.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 1 -- 172.21.15.107:0/2901998170 >> v1:172.21.15.107:6789/0 conn(0x7f72a413f710 legacy=0x7f72a4132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:03.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 1 -- 172.21.15.107:0/2901998170 shutdown_connections 2024-03-25T19:38:03.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 1 -- 172.21.15.107:0/2901998170 >> 172.21.15.107:0/2901998170 conn(0x7f72a40b7ea0 msgr2=0x7f72a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:03.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 1 -- 172.21.15.107:0/2901998170 shutdown_connections 2024-03-25T19:38:03.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 1 -- 172.21.15.107:0/2901998170 wait complete. 2024-03-25T19:38:03.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:03.901+0000 7f72a9a14640 1 librados: shutdown 2024-03-25T19:38:03.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:04.609 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:02:34.925 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:04.617 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:02:34.933 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:04.667 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:02:34.984 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:04.668 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:02:34.984 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:04.680 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:02:34.996 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:04.693 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:02:34.980 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:04.700 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:02:35.016 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:04.707 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:02:34.994 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:04.733 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:02:35.020 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:10.719 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.837+0000 7ff7c4a4a640 10 monclient: get_monmap_and_config 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.837+0000 7ff7c4a4a640 10 monclient: build_initial_monmap 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.837+0000 7ff7c4a4a640 10 monclient: monmap: 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:38:10.838661+0000 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:38:10.838661+0000 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:38:10.838 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:10.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.838+0000 7ff7c4a4a640 1 Processor -- start 2024-03-25T19:38:10.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.838+0000 7ff7c4a4a640 1 -- start start 2024-03-25T19:38:10.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.838+0000 7ff7c4a4a640 10 monclient: init 2024-03-25T19:38:10.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.839+0000 7ff7c4a4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:10.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.839+0000 7ff7c4a4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:10.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.839+0000 7ff7c4a4a640 10 monclient(hunting): picked mon.noname-b con 0x7ff7c00b0070 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:10.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.839+0000 7ff7c4a4a640 10 monclient(hunting): picked mon.noname-c con 0x7ff7c013c170 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:10.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.839+0000 7ff7c4a4a640 10 monclient(hunting): picked mon.noname-a con 0x7ff7c006fa90 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:10.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.839+0000 7ff7c4a4a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c00b0450 con 0x7ff7c006fa90 2024-03-25T19:38:10.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.839+0000 7ff7c4a4a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c0071440 con 0x7ff7c013c170 2024-03-25T19:38:10.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.839+0000 7ff7c4a4a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c00b0bf0 con 0x7ff7c00b0070 2024-03-25T19:38:10.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.839+0000 7ff7c4a4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:10.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.839+0000 7ff7c4a4a640 10 monclient(hunting): authenticate will time out at 4683.180353s 2024-03-25T19:38:10.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.845+0000 7ff7be59c640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff7c00b0070 0x7ff7c013ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:58688/0 (socket says 172.21.15.107:58688) 2024-03-25T19:38:10.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.845+0000 7ff7be59c640 1 -- 172.21.15.107:0/3213865038 learned_addr learned my addr 172.21.15.107:0/3213865038 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:10.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.856+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013000850 0 0) 0x7ff7c0071440 con 0x7ff7c013c170 2024-03-25T19:38:10.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.856+0000 7ff7bd59a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:10.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.856+0000 7ff7bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:10.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.856+0000 7ff7bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:10.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.856+0000 7ff7bd59a640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:38:10.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.856+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a8003520 con 0x7ff7c013c170 2024-03-25T19:38:10.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.861+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123897442 0 0) 0x7ff7c00b0bf0 con 0x7ff7c00b0070 2024-03-25T19:38:10.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.862+0000 7ff7bd59a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:10.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.862+0000 7ff7bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:10.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.862+0000 7ff7bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:10.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.862+0000 7ff7bd59a640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:38:10.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.862+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c0071440 con 0x7ff7c00b0070 2024-03-25T19:38:10.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.865+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217545129 0 0) 0x7ff7c00b0450 con 0x7ff7c006fa90 2024-03-25T19:38:10.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.865+0000 7ff7bd59a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:10.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.865+0000 7ff7bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:10.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.865+0000 7ff7bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:10.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.865+0000 7ff7bd59a640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:38:10.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.865+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c00b0bf0 con 0x7ff7c006fa90 2024-03-25T19:38:10.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3223138800 0 0) 0x7ff7a8003520 con 0x7ff7c013c170 2024-03-25T19:38:10.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 >> v1:172.21.15.107:6790/0 conn(0x7ff7c00b0070 legacy=0x7ff7c013ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:10.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 >> v1:172.21.15.107:6789/0 conn(0x7ff7c006fa90 legacy=0x7ff7c014a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:10.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 10 monclient: _finish_hunting 0 2024-03-25T19:38:10.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 1 monclient: found mon.noname-c 2024-03-25T19:38:10.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:10.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:38:10.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c00716b0 con 0x7ff7c013c170 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 10 monclient: _finish_auth 0 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 10 monclient: _check_auth_tickets 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff7b4001690 con 0x7ff7c013c170 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7c4a4a640 5 monclient: authenticate success, global_id 4315 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7c4a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7bd59a640 10 monclient: dump: 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:10.874 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:10.875 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:10.875 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:10.875 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:10.875 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:10.875 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.873+0000 7ff7c4a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:10.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.891+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7b40017f0 con 0x7ff7c013c170 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.891+0000 7ff7bd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.891+0000 7ff7bd59a640 1 -- 172.21.15.107:0/3213865038 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff7b4001c00 con 0x7ff7c013c170 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.891+0000 7ff7bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.891+0000 7ff7bd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.891+0000 7ff7bd59a640 10 monclient: dump: 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:10.892 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.892+0000 7ff7c4a4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.892+0000 7ff7c4a4a640 4 set_mon_vals no callback set 2024-03-25T19:38:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.892+0000 7ff7c4a4a640 10 monclient: shutdown 2024-03-25T19:38:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.892+0000 7ff7c4a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.892+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/3213865038 >> v1:172.21.15.195:6789/0 conn(0x7ff7c013c170 legacy=0x7ff7c0141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.893+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/3213865038 shutdown_connections 2024-03-25T19:38:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.893+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/3213865038 >> 172.21.15.107:0/3213865038 conn(0x7ff7c00b0ef0 msgr2=0x7ff7c00b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:10.894 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.893+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/3213865038 shutdown_connections 2024-03-25T19:38:10.894 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.893+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/3213865038 wait complete. 2024-03-25T19:38:10.894 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.894+0000 7ff7c4a4a640 10 monclient: build_initial_monmap 2024-03-25T19:38:10.894 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.894+0000 7ff7c4a4a640 10 monclient: monmap: 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.895+0000 7ff7c4a4a640 1 librados: starting msgr at 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.895+0000 7ff7c4a4a640 1 librados: starting objecter 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.895+0000 7ff7c4a4a640 1 Processor -- start 2024-03-25T19:38:10.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.895+0000 7ff7c4a4a640 1 -- start start 2024-03-25T19:38:10.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.895+0000 7ff7c4a4a640 1 librados: setting wanted keys 2024-03-25T19:38:10.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.895+0000 7ff7c4a4a640 1 librados: calling monclient init 2024-03-25T19:38:10.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.895+0000 7ff7c4a4a640 10 monclient: init 2024-03-25T19:38:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.896+0000 7ff7c4a4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.896+0000 7ff7c4a4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.896+0000 7ff7c4a4a640 10 monclient(hunting): picked mon.noname-a con 0x7ff7c006fa90 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.896+0000 7ff7c4a4a640 10 monclient(hunting): picked mon.noname-b con 0x7ff7c00b0070 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.896+0000 7ff7c4a4a640 10 monclient(hunting): picked mon.noname-c con 0x7ff7c013c170 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.896+0000 7ff7c4a4a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c012d400 con 0x7ff7c006fa90 2024-03-25T19:38:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.896+0000 7ff7c4a4a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c0132900 con 0x7ff7c00b0070 2024-03-25T19:38:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.896+0000 7ff7c4a4a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c0132be0 con 0x7ff7c013c170 2024-03-25T19:38:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.896+0000 7ff7c4a4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.896+0000 7ff7c4a4a640 10 monclient(hunting): authenticate will time out at 4683.237081s 2024-03-25T19:38:10.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.901+0000 7ff7bed9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff7c013c170 0x7ff7c01db470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:58726/0 (socket says 172.21.15.107:58726) 2024-03-25T19:38:10.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.901+0000 7ff7bed9d640 1 -- 172.21.15.107:0/2711255537 learned_addr learned my addr 172.21.15.107:0/2711255537 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:10.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.901+0000 7ff7aeffd640 10 client.?.objecter ms_handle_connect 0x7ff7c013c170 2024-03-25T19:38:10.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.901+0000 7ff7aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.913+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938080226 0 0) 0x7ff7c0132be0 con 0x7ff7c013c170 2024-03-25T19:38:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.913+0000 7ff7aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.913+0000 7ff7aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.913+0000 7ff7aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:10.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.913+0000 7ff7aeffd640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:38:10.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.913+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a0004eb0 con 0x7ff7c013c170 2024-03-25T19:38:10.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.915+0000 7ff7aeffd640 10 client.?.objecter ms_handle_connect 0x7ff7c00b0070 2024-03-25T19:38:10.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.915+0000 7ff7aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:10.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.921+0000 7ff7aeffd640 10 client.?.objecter ms_handle_connect 0x7ff7c006fa90 2024-03-25T19:38:10.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.921+0000 7ff7aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:10.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.930+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742090173 0 0) 0x7ff7c0132900 con 0x7ff7c00b0070 2024-03-25T19:38:10.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.930+0000 7ff7aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:10.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.930+0000 7ff7aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:10.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.930+0000 7ff7aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:10.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.930+0000 7ff7aeffd640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:38:10.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.930+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c0132be0 con 0x7ff7c00b0070 2024-03-25T19:38:10.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.932+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598799747 0 0) 0x7ff7c0071440 con 0x7ff7c006fa90 2024-03-25T19:38:10.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.932+0000 7ff7aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:10.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.932+0000 7ff7aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:10.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.932+0000 7ff7aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:10.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.932+0000 7ff7aeffd640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:38:10.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.932+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c0132900 con 0x7ff7c006fa90 2024-03-25T19:38:10.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.941+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 626608557 0 0) 0x7ff7a0004eb0 con 0x7ff7c013c170 2024-03-25T19:38:10.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.942+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7c0071440 con 0x7ff7c013c170 2024-03-25T19:38:10.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.942+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff7b8001d10 con 0x7ff7c013c170 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.942+0000 7ff7aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.942+0000 7ff7aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.942+0000 7ff7aeffd640 10 monclient(hunting): dump: 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:10.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.942+0000 7ff7aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.945+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3719589852 0 0) 0x7ff7c0132900 con 0x7ff7c006fa90 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.945+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7a0004eb0 con 0x7ff7c006fa90 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.945+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff7b0001790 con 0x7ff7c006fa90 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.945+0000 7ff7aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.945+0000 7ff7aeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.945+0000 7ff7aeffd640 10 monclient(hunting): dump: 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:10.945 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:10.946 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:10.946 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:10.946 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:10.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.948+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4147387100 0 0) 0x7ff7c0132be0 con 0x7ff7c00b0070 2024-03-25T19:38:10.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.948+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7c0132900 con 0x7ff7c00b0070 2024-03-25T19:38:10.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.948+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff7b4001b00 con 0x7ff7c00b0070 2024-03-25T19:38:10.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.948+0000 7ff7aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:10.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.948+0000 7ff7aeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:10.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.948+0000 7ff7aeffd640 10 monclient(hunting): dump: 2024-03-25T19:38:10.948 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:10.948 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:10.948 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:10.949 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:10.949 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:10.949 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:10.949 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:10.949 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:10.949 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:10.949 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:10.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3557976880 0 0) 0x7ff7c0071440 con 0x7ff7c013c170 2024-03-25T19:38:10.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 >> v1:172.21.15.195:6789/0 conn(0x7ff7c00b0070 legacy=0x7ff7c01dad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:10.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 >> v1:172.21.15.107:6789/0 conn(0x7ff7c006fa90 legacy=0x7ff7c0140300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7aeffd640 1 monclient: found mon.c 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7aeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c0141420 con 0x7ff7c013c170 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7c4a4a640 5 monclient: authenticate success, global_id 4331 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7c4a4a640 10 monclient: _renew_subs 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7c4a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/2711255537 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c0140ff0 con 0x7ff7c013c170 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7c4a4a640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7c4a4a640 10 monclient: _renew_subs 2024-03-25T19:38:10.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7c4a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:10.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/2711255537 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c01e32e0 con 0x7ff7c013c170 2024-03-25T19:38:10.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.949+0000 7ff7c4a4a640 1 librados: init done 2024-03-25T19:38:10.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.950+0000 7ff7c4a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:10.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.950+0000 7ff7c4a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:10.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.950+0000 7ff7c4a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:10.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:10.950+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/2711255537 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c00627b0 con 0x7ff7c013c170 2024-03-25T19:38:11.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.174+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7b8003990 con 0x7ff7c013c170 2024-03-25T19:38:11.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.174+0000 7ff7aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:11.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.174+0000 7ff7bcd99640 4 set_mon_vals no callback set 2024-03-25T19:38:11.261 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.261+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff7b8003d60 con 0x7ff7c013c170 2024-03-25T19:38:11.261 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.261+0000 7ff7aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:11.261 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.261+0000 7ff7aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:11.261 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.261+0000 7ff7aeffd640 10 monclient: dump: 2024-03-25T19:38:11.261 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:11.261 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:11.261 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:11.261 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:11.261 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:11.261 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:11.262 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:11.262 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:11.262 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:11.262 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:11.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.262+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7ff7b8003f40 con 0x7ff7c013c170 2024-03-25T19:38:11.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.263+0000 7ff7be59c640 10 monclient: get_auth_request con 0x7ff7a006e0b0 auth_method 0 2024-03-25T19:38:11.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.263+0000 7ff7aeffd640 10 client.4331.objecter ms_handle_connect 0x7ff7a006e0b0 2024-03-25T19:38:11.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.269+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7ff7b8054b90 con 0x7ff7c013c170 2024-03-25T19:38:11.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.269+0000 7ff7aeffd640 10 client.4331.objecter ms_dispatch 0x7ff7c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:11.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.269+0000 7ff7aeffd640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:11.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.269+0000 7ff7aeffd640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:11.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.269+0000 7ff7aeffd640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-25T19:38:11.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.269+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7b80506a0 con 0x7ff7c013c170 2024-03-25T19:38:11.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.269+0000 7ff7aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:11.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.269+0000 7ff7aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:11.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.536+0000 7ff7c4a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:11.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.537+0000 7ff7c4a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:11.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.537+0000 7ff7c4a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:11.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.537+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/2711255537 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7c00ae550 con 0x7ff7c013c170 2024-03-25T19:38:11.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.581+0000 7ff7aeffd640 1 -- 172.21.15.107:0/2711255537 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7ff7b80525d0 con 0x7ff7c013c170 2024-03-25T19:38:11.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.581+0000 7ff7aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:11.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.581+0000 7ff7aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:11.581 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:38:11.581 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:11.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.583+0000 7ff7c4a4a640 10 librados: watch_flush enter 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.583+0000 7ff7c4a4a640 10 librados: watch_flush exit 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.583+0000 7ff7c4a4a640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.583+0000 7ff7c4a4a640 10 client.4331.objecter successfully canceled tick 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.584+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/2711255537 >> v1:172.21.15.195:6800/2984408421 conn(0x7ff7a006e0b0 legacy=0x7ff7a0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.584+0000 7ff7c4a4a640 10 monclient: shutdown 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.584+0000 7ff7c4a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.584+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/2711255537 >> v1:172.21.15.107:6790/0 conn(0x7ff7c013c170 legacy=0x7ff7c01db470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.584+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/2711255537 shutdown_connections 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.584+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/2711255537 >> 172.21.15.107:0/2711255537 conn(0x7ff7c00b0ef0 msgr2=0x7ff7c006ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.584+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/2711255537 shutdown_connections 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.584+0000 7ff7c4a4a640 1 -- 172.21.15.107:0/2711255537 wait complete. 2024-03-25T19:38:11.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:11.584+0000 7ff7c4a4a640 1 librados: shutdown 2024-03-25T19:38:11.594 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:18.401 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:18.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.517+0000 7f20091c8640 10 monclient: get_monmap_and_config 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.517+0000 7f20091c8640 10 monclient: build_initial_monmap 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.517+0000 7f20091c8640 10 monclient: monmap: 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:38:18.518243+0000 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:38:18.518243+0000 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:38:18.517 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:18.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 1 Processor -- start 2024-03-25T19:38:18.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 1 -- start start 2024-03-25T19:38:18.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 10 monclient: init 2024-03-25T19:38:18.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:18.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:18.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 10 monclient(hunting): picked mon.noname-b con 0x7f20040a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:18.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 10 monclient(hunting): picked mon.noname-c con 0x7f200413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:18.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 10 monclient(hunting): picked mon.noname-a con 0x7f20041430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:18.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2004142520 con 0x7f20041430c0 2024-03-25T19:38:18.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20040b28a0 con 0x7f200413f710 2024-03-25T19:38:18.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20040b2a30 con 0x7f20040a98f0 2024-03-25T19:38:18.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:18.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.518+0000 7f20091c8640 10 monclient(hunting): authenticate will time out at 4690.859646s 2024-03-25T19:38:18.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.524+0000 7f200259c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f200413f710 0x7f20041429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:34926/0 (socket says 172.21.15.107:34926) 2024-03-25T19:38:18.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.524+0000 7f200259c640 1 -- 172.21.15.107:0/587475917 learned_addr learned my addr 172.21.15.107:0/587475917 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:18.531 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.531+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476678223 0 0) 0x7f20040b28a0 con 0x7f200413f710 2024-03-25T19:38:18.531 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.531+0000 7f2001d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:18.531 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.531+0000 7f2001d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:18.531 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.531+0000 7f2001d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:18.531 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.531+0000 7f2001d9b640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:38:18.531 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.531+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fec003520 con 0x7f200413f710 2024-03-25T19:38:18.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734663070 0 0) 0x7f20040b2a30 con 0x7f20040a98f0 2024-03-25T19:38:18.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:18.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:18.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:18.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:38:18.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20040b28a0 con 0x7f20040a98f0 2024-03-25T19:38:18.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478724577 0 0) 0x7f2004142520 con 0x7f20041430c0 2024-03-25T19:38:18.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:18.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:18.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:18.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:38:18.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.538+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20040b2a30 con 0x7f20041430c0 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069241527 0 0) 0x7f1fec003520 con 0x7f200413f710 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 >> v1:172.21.15.107:6790/0 conn(0x7f20040a98f0 legacy=0x7f20040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 >> v1:172.21.15.107:6789/0 conn(0x7f20041430c0 legacy=0x7f200414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 1 monclient: found mon.noname-c 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20040b44c0 con 0x7f200413f710 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 10 monclient: _finish_auth 0 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1ff8001690 con 0x7f200413f710 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:18.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f2001d9b640 10 monclient: dump: 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f20091c8640 5 monclient: authenticate success, global_id 4321 2024-03-25T19:38:18.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.541+0000 7f20091c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:18.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.553+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ff80017f0 con 0x7f200413f710 2024-03-25T19:38:18.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.553+0000 7f2001d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.553+0000 7f2001d9b640 1 -- 172.21.15.107:0/587475917 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1ff8001c00 con 0x7f200413f710 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.553+0000 7f2001d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.553+0000 7f2001d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.553+0000 7f2001d9b640 10 monclient: dump: 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:18.553 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:18.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.553+0000 7f20091c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:18.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.553+0000 7f20091c8640 4 set_mon_vals no callback set 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.554+0000 7f20091c8640 10 monclient: shutdown 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.554+0000 7f20091c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.554+0000 7f20091c8640 1 -- 172.21.15.107:0/587475917 >> v1:172.21.15.195:6789/0 conn(0x7f200413f710 legacy=0x7f20041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.554+0000 7f20091c8640 1 -- 172.21.15.107:0/587475917 shutdown_connections 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.554+0000 7f20091c8640 1 -- 172.21.15.107:0/587475917 >> 172.21.15.107:0/587475917 conn(0x7f20040b7ea0 msgr2=0x7f20040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.554+0000 7f20091c8640 1 -- 172.21.15.107:0/587475917 shutdown_connections 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.554+0000 7f20091c8640 1 -- 172.21.15.107:0/587475917 wait complete. 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.555+0000 7f20091c8640 10 monclient: build_initial_monmap 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.555+0000 7f20091c8640 10 monclient: monmap: 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:38:18.555 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:38:18.556 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:18.556 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:38:18.556 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:18.556 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:38:18.556 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:38:18.556 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:18.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.555+0000 7f20091c8640 1 librados: starting msgr at 2024-03-25T19:38:18.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.555+0000 7f20091c8640 1 librados: starting objecter 2024-03-25T19:38:18.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 1 Processor -- start 2024-03-25T19:38:18.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 1 -- start start 2024-03-25T19:38:18.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 1 librados: setting wanted keys 2024-03-25T19:38:18.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 1 librados: calling monclient init 2024-03-25T19:38:18.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 10 monclient: init 2024-03-25T19:38:18.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:18.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:18.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 10 monclient(hunting): picked mon.noname-c con 0x7f20040a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:18.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 10 monclient(hunting): picked mon.noname-b con 0x7f200413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:18.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 10 monclient(hunting): picked mon.noname-a con 0x7f20041430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:18.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2004138250 con 0x7f20041430c0 2024-03-25T19:38:18.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2004137d50 con 0x7f200413f710 2024-03-25T19:38:18.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2004137ee0 con 0x7f20040a98f0 2024-03-25T19:38:18.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:18.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.556+0000 7f20091c8640 10 monclient(hunting): authenticate will time out at 4690.897587s 2024-03-25T19:38:18.565 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.566+0000 7f2002d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f20040a98f0 0x7f2004137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35468/0 (socket says 172.21.15.107:35468) 2024-03-25T19:38:18.566 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.566+0000 7f2002d9d640 1 -- 172.21.15.107:0/600847955 learned_addr learned my addr 172.21.15.107:0/600847955 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:18.568 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.568+0000 7f1feb7fe640 10 client.?.objecter ms_handle_connect 0x7f20040a98f0 2024-03-25T19:38:18.568 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.568+0000 7f1feb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:18.572 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.573+0000 7f1feb7fe640 10 client.?.objecter ms_handle_connect 0x7f20041430c0 2024-03-25T19:38:18.572 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.573+0000 7f1feb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:18.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.574+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356265326 0 0) 0x7f2004137ee0 con 0x7f20040a98f0 2024-03-25T19:38:18.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.574+0000 7f1feb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:18.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.574+0000 7f1feb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:18.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.574+0000 7f1feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:18.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.574+0000 7f1feb7fe640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:38:18.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.574+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff0004eb0 con 0x7f20040a98f0 2024-03-25T19:38:18.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.579+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017312537 0 0) 0x7f2004138250 con 0x7f20041430c0 2024-03-25T19:38:18.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.579+0000 7f1feb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:18.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.579+0000 7f1feb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:18.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.579+0000 7f1feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:18.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.579+0000 7f1feb7fe640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:38:18.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.579+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2004137ee0 con 0x7f20041430c0 2024-03-25T19:38:18.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.580+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1008501664 0 0) 0x7f1ff0004eb0 con 0x7f20040a98f0 2024-03-25T19:38:18.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.581+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2004138250 con 0x7f20040a98f0 2024-03-25T19:38:18.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.581+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1ff4001a30 con 0x7f20040a98f0 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.581+0000 7f1feb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.581+0000 7f1feb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.581+0000 7f1feb7fe640 10 monclient(hunting): dump: 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.581+0000 7f1feb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.581+0000 7f1feb7fe640 10 client.?.objecter ms_handle_connect 0x7f200413f710 2024-03-25T19:38:18.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.581+0000 7f1feb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:18.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.586+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1061791651 0 0) 0x7f2004137ee0 con 0x7f20041430c0 2024-03-25T19:38:18.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.586+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ff0004eb0 con 0x7f20041430c0 2024-03-25T19:38:18.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.586+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1ffc001bd0 con 0x7f20041430c0 2024-03-25T19:38:18.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.586+0000 7f1feb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:18.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.586+0000 7f1feb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:18.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.586+0000 7f1feb7fe640 10 monclient(hunting): dump: 2024-03-25T19:38:18.586 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:18.586 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:18.586 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:18.586 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:18.587 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:18.587 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:18.587 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:18.587 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:18.587 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:18.587 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:18.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2225833734 0 0) 0x7f2004138250 con 0x7f20040a98f0 2024-03-25T19:38:18.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 >> v1:172.21.15.195:6789/0 conn(0x7f200413f710 legacy=0x7f20041d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:18.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 >> v1:172.21.15.107:6789/0 conn(0x7f20041430c0 legacy=0x7f20041d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:18.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f1feb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:18.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f1feb7fe640 1 monclient: found mon.c 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f1feb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f1feb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20040b7730 con 0x7f20040a98f0 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f1feb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f1feb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f1feb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f20091c8640 5 monclient: authenticate success, global_id 4337 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f20091c8640 10 monclient: _renew_subs 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f20091c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f20091c8640 1 -- 172.21.15.107:0/600847955 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20040b7920 con 0x7f20040a98f0 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f20091c8640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f20091c8640 10 monclient: _renew_subs 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f20091c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f20091c8640 1 -- 172.21.15.107:0/600847955 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20041e33d0 con 0x7f20040a98f0 2024-03-25T19:38:18.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.588+0000 7f20091c8640 1 librados: init done 2024-03-25T19:38:18.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.589+0000 7f20091c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:18.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.589+0000 7f20091c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:18.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.589+0000 7f20091c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:18.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.589+0000 7f20091c8640 1 -- 172.21.15.107:0/600847955 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20040627b0 con 0x7f20040a98f0 2024-03-25T19:38:18.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.918+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ff4001c70 con 0x7f20040a98f0 2024-03-25T19:38:18.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.918+0000 7f1feb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.918+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1ff4003a60 con 0x7f20040a98f0 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.918+0000 7f1feb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.918+0000 7f1feb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.918+0000 7f1feb7fe640 10 monclient: dump: 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:18.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.918+0000 7f200159a640 4 set_mon_vals no callback set 2024-03-25T19:38:18.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.918+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f1ff4003c40 con 0x7f20040a98f0 2024-03-25T19:38:18.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.919+0000 7f200259c640 10 monclient: get_auth_request con 0x7f1ff006de40 auth_method 0 2024-03-25T19:38:18.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.919+0000 7f1feb7fe640 10 client.4337.objecter ms_handle_connect 0x7f1ff006de40 2024-03-25T19:38:18.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.925+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f1ff40548d0 con 0x7f20040a98f0 2024-03-25T19:38:18.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.925+0000 7f1feb7fe640 10 client.4337.objecter ms_dispatch 0x7f2004000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:18.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.925+0000 7f1feb7fe640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:18.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.925+0000 7f1feb7fe640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:18.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.925+0000 7f1feb7fe640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-25T19:38:18.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.925+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ff40503e0 con 0x7f20040a98f0 2024-03-25T19:38:18.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.925+0000 7f1feb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:18.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:18.925+0000 7f1feb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:19.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.179+0000 7f20091c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:19.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.179+0000 7f20091c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:19.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.179+0000 7f20091c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:19.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.180+0000 7f20091c8640 1 -- 172.21.15.107:0/600847955 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f200413df50 con 0x7f20040a98f0 2024-03-25T19:38:19.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.205+0000 7f1feb7fe640 1 -- 172.21.15.107:0/600847955 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f1ff4052310 con 0x7f20040a98f0 2024-03-25T19:38:19.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.205+0000 7f1feb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:19.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.205+0000 7f1feb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:19.205 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:38:19.205 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:19.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.207+0000 7f20091c8640 10 librados: watch_flush enter 2024-03-25T19:38:19.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.207+0000 7f20091c8640 10 librados: watch_flush exit 2024-03-25T19:38:19.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.207+0000 7f20091c8640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-25T19:38:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.207+0000 7f20091c8640 10 client.4337.objecter successfully canceled tick 2024-03-25T19:38:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.208+0000 7f20091c8640 1 -- 172.21.15.107:0/600847955 >> v1:172.21.15.195:6800/2984408421 conn(0x7f1ff006de40 legacy=0x7f1ff0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.208+0000 7f20091c8640 10 monclient: shutdown 2024-03-25T19:38:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.208+0000 7f20091c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.208+0000 7f20091c8640 1 -- 172.21.15.107:0/600847955 >> v1:172.21.15.107:6790/0 conn(0x7f20040a98f0 legacy=0x7f2004137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.208+0000 7f20091c8640 1 -- 172.21.15.107:0/600847955 shutdown_connections 2024-03-25T19:38:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.208+0000 7f20091c8640 1 -- 172.21.15.107:0/600847955 >> 172.21.15.107:0/600847955 conn(0x7f20040b7ea0 msgr2=0x7f2004074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.208+0000 7f20091c8640 1 -- 172.21.15.107:0/600847955 shutdown_connections 2024-03-25T19:38:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.208+0000 7f20091c8640 1 -- 172.21.15.107:0/600847955 wait complete. 2024-03-25T19:38:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:19.208+0000 7f20091c8640 1 librados: shutdown 2024-03-25T19:38:19.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:24.650 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:02:54.966 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:24.655 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:02:54.971 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:24.691 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:02:55.007 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:24.702 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:02:55.018 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:24.712 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:02:54.999 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:24.714 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:02:55.031 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:24.721 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:02:55.008 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:24.748 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:02:55.064 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:24.751 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:02:55.037 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:26.025 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:26.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.138+0000 7f7c7e725640 10 monclient: get_monmap_and_config 2024-03-25T19:38:26.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.138+0000 7f7c7e725640 10 monclient: build_initial_monmap 2024-03-25T19:38:26.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.138+0000 7f7c7e725640 10 monclient: monmap: 2024-03-25T19:38:26.138 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:26.138 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:26.138 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:38:26.139532+0000 2024-03-25T19:38:26.138 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:38:26.139532+0000 2024-03-25T19:38:26.138 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:26.138 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:26.138 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:26.138 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:38:26.138 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:38:26.139 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:26.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 1 Processor -- start 2024-03-25T19:38:26.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 1 -- start start 2024-03-25T19:38:26.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 10 monclient: init 2024-03-25T19:38:26.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:26.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:26.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 10 monclient(hunting): picked mon.noname-a con 0x7f7c780b4f40 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:26.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 10 monclient(hunting): picked mon.noname-b con 0x7f7c7813fd90 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:26.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 10 monclient(hunting): picked mon.noname-c con 0x7f7c780b4540 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:26.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c780aa0c0 con 0x7f7c780b4f40 2024-03-25T19:38:26.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c780ab170 con 0x7f7c780b4540 2024-03-25T19:38:26.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c780a9ba0 con 0x7f7c7813fd90 2024-03-25T19:38:26.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:26.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.139+0000 7f7c7e725640 10 monclient(hunting): authenticate will time out at 4698.480676s 2024-03-25T19:38:26.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.140+0000 7f7c77fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7c780b4f40 0x7f7c7813f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51344/0 (socket says 172.21.15.107:51344) 2024-03-25T19:38:26.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.140+0000 7f7c77fff640 1 -- 172.21.15.107:0/3844957862 learned_addr learned my addr 172.21.15.107:0/3844957862 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:26.198 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.199+0000 7f7c76ffd640 1 -- 172.21.15.107:0/3844957862 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453476079 0 0) 0x7f7c780a9ba0 con 0x7f7c7813fd90 2024-03-25T19:38:26.198 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.199+0000 7f7c76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:26.198 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.199+0000 7f7c76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:26.199 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.199+0000 7f7c76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:26.199 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.199+0000 7f7c76ffd640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:38:26.199 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.199+0000 7f7c76ffd640 1 -- 172.21.15.107:0/3844957862 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c60003520 con 0x7f7c7813fd90 2024-03-25T19:38:26.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.209+0000 7f7c76ffd640 1 -- 172.21.15.107:0/3844957862 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 757727259 0 0) 0x7f7c60003520 con 0x7f7c7813fd90 2024-03-25T19:38:26.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 1 -- 172.21.15.107:0/3844957862 >> v1:172.21.15.195:6789/0 conn(0x7f7c780b4540 legacy=0x7f7c7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 1 -- 172.21.15.107:0/3844957862 >> v1:172.21.15.107:6789/0 conn(0x7f7c780b4f40 legacy=0x7f7c7813f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 1 monclient: found mon.noname-b 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 1 -- 172.21.15.107:0/3844957862 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c780a9680 con 0x7f7c7813fd90 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 1 -- 172.21.15.107:0/3844957862 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7c68001690 con 0x7f7c7813fd90 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:26.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c76ffd640 10 monclient: dump: 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c7e725640 5 monclient: authenticate success, global_id 4340 2024-03-25T19:38:26.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.210+0000 7f7c7e725640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:26.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.219+0000 7f7c76ffd640 1 -- 172.21.15.107:0/3844957862 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c680017f0 con 0x7f7c7813fd90 2024-03-25T19:38:26.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.219+0000 7f7c76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:26.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.219+0000 7f7c76ffd640 1 -- 172.21.15.107:0/3844957862 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7c68001c00 con 0x7f7c7813fd90 2024-03-25T19:38:26.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.219+0000 7f7c7e725640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:26.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.219+0000 7f7c7e725640 4 set_mon_vals no callback set 2024-03-25T19:38:26.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.219+0000 7f7c76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:26.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.219+0000 7f7c76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:26.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.219+0000 7f7c76ffd640 10 monclient: dump: 2024-03-25T19:38:26.219 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:26.219 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:26.219 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:26.220 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:26.220 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:26.220 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:26.220 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:26.220 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:26.220 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:26.220 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:26.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.220+0000 7f7c7e725640 10 monclient: shutdown 2024-03-25T19:38:26.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.220+0000 7f7c7e725640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:26.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.220+0000 7f7c7e725640 1 -- 172.21.15.107:0/3844957862 >> v1:172.21.15.107:6790/0 conn(0x7f7c7813fd90 legacy=0x7f7c781423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:26.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.220+0000 7f7c7e725640 1 -- 172.21.15.107:0/3844957862 shutdown_connections 2024-03-25T19:38:26.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.220+0000 7f7c7e725640 1 -- 172.21.15.107:0/3844957862 >> 172.21.15.107:0/3844957862 conn(0x7f7c780617a0 msgr2=0x7f7c78061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.220+0000 7f7c7e725640 1 -- 172.21.15.107:0/3844957862 shutdown_connections 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.220+0000 7f7c7e725640 1 -- 172.21.15.107:0/3844957862 wait complete. 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.221+0000 7f7c7e725640 10 monclient: build_initial_monmap 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.221+0000 7f7c7e725640 10 monclient: monmap: 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.221+0000 7f7c7e725640 1 librados: starting msgr at 2024-03-25T19:38:26.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.221+0000 7f7c7e725640 1 librados: starting objecter 2024-03-25T19:38:26.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.221+0000 7f7c7e725640 1 Processor -- start 2024-03-25T19:38:26.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 1 -- start start 2024-03-25T19:38:26.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 1 librados: setting wanted keys 2024-03-25T19:38:26.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 1 librados: calling monclient init 2024-03-25T19:38:26.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 10 monclient: init 2024-03-25T19:38:26.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:26.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:26.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 10 monclient(hunting): picked mon.noname-b con 0x7f7c780b4540 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:26.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 10 monclient(hunting): picked mon.noname-a con 0x7f7c780b4f40 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:26.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 10 monclient(hunting): picked mon.noname-c con 0x7f7c7813fd90 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:26.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c781d7be0 con 0x7f7c780b4f40 2024-03-25T19:38:26.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c780b06e0 con 0x7f7c780b4540 2024-03-25T19:38:26.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c781d8780 con 0x7f7c7813fd90 2024-03-25T19:38:26.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:26.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.222+0000 7f7c7e725640 10 monclient(hunting): authenticate will time out at 4698.563545s 2024-03-25T19:38:26.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.229+0000 7f7c7ccc2640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f7c7813fd90 0x7f7c78137950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34362/0 (socket says 172.21.15.107:34362) 2024-03-25T19:38:26.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.229+0000 7f7c7ccc2640 1 -- 172.21.15.107:0/1784940220 learned_addr learned my addr 172.21.15.107:0/1784940220 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:26.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.231+0000 7f7c74ff9640 10 client.?.objecter ms_handle_connect 0x7f7c7813fd90 2024-03-25T19:38:26.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.231+0000 7f7c74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:26.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.244+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960062700 0 0) 0x7f7c781d8780 con 0x7f7c7813fd90 2024-03-25T19:38:26.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.244+0000 7f7c74ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:26.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.244+0000 7f7c74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:26.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.244+0000 7f7c74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:26.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.244+0000 7f7c74ff9640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:38:26.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.244+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c64004eb0 con 0x7f7c7813fd90 2024-03-25T19:38:26.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.248+0000 7f7c74ff9640 10 client.?.objecter ms_handle_connect 0x7f7c780b4f40 2024-03-25T19:38:26.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.248+0000 7f7c74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:26.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.253+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435957871 0 0) 0x7f7c781d7be0 con 0x7f7c780b4f40 2024-03-25T19:38:26.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.253+0000 7f7c74ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:26.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.253+0000 7f7c74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:26.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.254+0000 7f7c74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:26.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.254+0000 7f7c74ff9640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:38:26.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.254+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c781d8780 con 0x7f7c780b4f40 2024-03-25T19:38:26.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.254+0000 7f7c74ff9640 10 client.?.objecter ms_handle_connect 0x7f7c780b4540 2024-03-25T19:38:26.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.254+0000 7f7c74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:26.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.259+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 358985486 0 0) 0x7f7c64004eb0 con 0x7f7c7813fd90 2024-03-25T19:38:26.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.259+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c781d7be0 con 0x7f7c7813fd90 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.259+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7c70001c50 con 0x7f7c7813fd90 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.259+0000 7f7c74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.259+0000 7f7c74ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.259+0000 7f7c74ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:26.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.259+0000 7f7c74ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:26.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.263+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697786418 0 0) 0x7f7c780b06e0 con 0x7f7c780b4540 2024-03-25T19:38:26.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.263+0000 7f7c74ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:26.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.263+0000 7f7c74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:26.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.263+0000 7f7c74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:26.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.263+0000 7f7c74ff9640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:38:26.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.263+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c64004eb0 con 0x7f7c780b4540 2024-03-25T19:38:26.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.269+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3563933303 0 0) 0x7f7c781d8780 con 0x7f7c780b4f40 2024-03-25T19:38:26.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.269+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c780b06e0 con 0x7f7c780b4f40 2024-03-25T19:38:26.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.269+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7c68001bc0 con 0x7f7c780b4f40 2024-03-25T19:38:26.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.269+0000 7f7c74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:26.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.269+0000 7f7c74ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:26.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.269+0000 7f7c74ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:26.269 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:26.269 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:26.269 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:26.269 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:26.269 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:26.270 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:26.270 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:26.270 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:26.270 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:26.270 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:26.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3397875213 0 0) 0x7f7c781d7be0 con 0x7f7c7813fd90 2024-03-25T19:38:26.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 >> v1:172.21.15.195:6789/0 conn(0x7f7c780b4540 legacy=0x7f7c781d8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:26.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 >> v1:172.21.15.107:6789/0 conn(0x7f7c780b4f40 legacy=0x7f7c78137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:26.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c74ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:38:26.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c74ff9640 1 monclient: found mon.c 2024-03-25T19:38:26.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c74ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:26.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c74ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:26.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c78138060 con 0x7f7c7813fd90 2024-03-25T19:38:26.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c74ff9640 10 monclient: _finish_auth 0 2024-03-25T19:38:26.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c74ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:38:26.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:26.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c7e725640 5 monclient: authenticate success, global_id 4343 2024-03-25T19:38:26.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c7e725640 10 monclient: _renew_subs 2024-03-25T19:38:26.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c7e725640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:26.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c7e725640 1 -- 172.21.15.107:0/1784940220 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c780b09b0 con 0x7f7c7813fd90 2024-03-25T19:38:26.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c7e725640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:26.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c7e725640 10 monclient: _renew_subs 2024-03-25T19:38:26.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c7e725640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:26.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c7e725640 1 -- 172.21.15.107:0/1784940220 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c781384c0 con 0x7f7c7813fd90 2024-03-25T19:38:26.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.272+0000 7f7c7e725640 1 librados: init done 2024-03-25T19:38:26.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.273+0000 7f7c7e725640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:26.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.273+0000 7f7c7e725640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:26.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.273+0000 7f7c7e725640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:26.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.273+0000 7f7c7e725640 1 -- 172.21.15.107:0/1784940220 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c780627b0 con 0x7f7c7813fd90 2024-03-25T19:38:26.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.591+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c70003990 con 0x7f7c7813fd90 2024-03-25T19:38:26.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.591+0000 7f7c74ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.591+0000 7f7c767fc640 4 set_mon_vals no callback set 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.591+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f7c70003d20 con 0x7f7c7813fd90 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.591+0000 7f7c74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.591+0000 7f7c74ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.591+0000 7f7c74ff9640 10 monclient: dump: 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:26.591 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:26.592 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:26.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.591+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f7c70003f00 con 0x7f7c7813fd90 2024-03-25T19:38:26.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.592+0000 7f7c77fff640 10 monclient: get_auth_request con 0x7f7c6406df10 auth_method 0 2024-03-25T19:38:26.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.592+0000 7f7c74ff9640 10 client.4343.objecter ms_handle_connect 0x7f7c6406df10 2024-03-25T19:38:26.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.597+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f7c70054ad0 con 0x7f7c7813fd90 2024-03-25T19:38:26.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.597+0000 7f7c74ff9640 10 client.4343.objecter ms_dispatch 0x7f7c78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:26.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.597+0000 7f7c74ff9640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:26.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.597+0000 7f7c74ff9640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:26.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.598+0000 7f7c74ff9640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-25T19:38:26.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.598+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c700505e0 con 0x7f7c7813fd90 2024-03-25T19:38:26.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.598+0000 7f7c74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:26.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.598+0000 7f7c74ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:26.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.844+0000 7f7c7e725640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:26.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.844+0000 7f7c7e725640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:26.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.844+0000 7f7c7e725640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:26.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.844+0000 7f7c7e725640 1 -- 172.21.15.107:0/1784940220 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c7813aad0 con 0x7f7c7813fd90 2024-03-25T19:38:26.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.869+0000 7f7c74ff9640 1 -- 172.21.15.107:0/1784940220 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f7c70052510 con 0x7f7c7813fd90 2024-03-25T19:38:26.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.870+0000 7f7c74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:26.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.870+0000 7f7c74ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:26.869 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:38:26.870 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.872+0000 7f7c7e725640 10 librados: watch_flush enter 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.872+0000 7f7c7e725640 10 librados: watch_flush exit 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.872+0000 7f7c7e725640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.872+0000 7f7c7e725640 10 client.4343.objecter successfully canceled tick 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.873+0000 7f7c7e725640 1 -- 172.21.15.107:0/1784940220 >> v1:172.21.15.195:6800/2984408421 conn(0x7f7c6406df10 legacy=0x7f7c640703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.873+0000 7f7c7e725640 10 monclient: shutdown 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.873+0000 7f7c7e725640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.873+0000 7f7c7e725640 1 -- 172.21.15.107:0/1784940220 >> v1:172.21.15.107:6790/0 conn(0x7f7c7813fd90 legacy=0x7f7c78137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.873+0000 7f7c7e725640 1 -- 172.21.15.107:0/1784940220 shutdown_connections 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.873+0000 7f7c7e725640 1 -- 172.21.15.107:0/1784940220 >> 172.21.15.107:0/1784940220 conn(0x7f7c780617a0 msgr2=0x7f7c78139d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.873+0000 7f7c7e725640 1 -- 172.21.15.107:0/1784940220 shutdown_connections 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.873+0000 7f7c7e725640 1 -- 172.21.15.107:0/1784940220 wait complete. 2024-03-25T19:38:26.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:26.873+0000 7f7c7e725640 1 librados: shutdown 2024-03-25T19:38:26.885 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:33.691 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:33.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.806+0000 7f353e120640 10 monclient: get_monmap_and_config 2024-03-25T19:38:33.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.806+0000 7f353e120640 10 monclient: build_initial_monmap 2024-03-25T19:38:33.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.807+0000 7f353e120640 10 monclient: monmap: 2024-03-25T19:38:33.806 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:33.806 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:33.806 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:38:33.807940+0000 2024-03-25T19:38:33.806 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:38:33.807940+0000 2024-03-25T19:38:33.806 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:33.807 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:33.807 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:33.807 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:38:33.807 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:38:33.807 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:33.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.807+0000 7f353e120640 1 Processor -- start 2024-03-25T19:38:33.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.807+0000 7f353e120640 1 -- start start 2024-03-25T19:38:33.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.807+0000 7f353e120640 10 monclient: init 2024-03-25T19:38:33.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.808+0000 7f353e120640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:33.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.808+0000 7f353e120640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:33.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.808+0000 7f353e120640 10 monclient(hunting): picked mon.noname-a con 0x7f35380a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:33.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.808+0000 7f353e120640 10 monclient(hunting): picked mon.noname-c con 0x7f353813f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:33.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.808+0000 7f353e120640 10 monclient(hunting): picked mon.noname-b con 0x7f35381430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:33.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.808+0000 7f353e120640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3538142520 con 0x7f35380a98f0 2024-03-25T19:38:33.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.808+0000 7f353e120640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35380b28a0 con 0x7f353813f710 2024-03-25T19:38:33.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.808+0000 7f353e120640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35380b2a30 con 0x7f35381430c0 2024-03-25T19:38:33.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.808+0000 7f353e120640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:33.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.808+0000 7f353e120640 10 monclient(hunting): authenticate will time out at 4706.149341s 2024-03-25T19:38:33.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.814+0000 7f3537fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f35381430c0 0x7f353814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44630/0 (socket says 172.21.15.107:44630) 2024-03-25T19:38:33.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.814+0000 7f3537fff640 1 -- 172.21.15.107:0/3639837683 learned_addr learned my addr 172.21.15.107:0/3639837683 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:33.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.826+0000 7f35367fc640 1 -- 172.21.15.107:0/3639837683 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417139344 0 0) 0x7f35380b2a30 con 0x7f35381430c0 2024-03-25T19:38:33.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.826+0000 7f35367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:33.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.826+0000 7f35367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:33.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.826+0000 7f35367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:33.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.826+0000 7f35367fc640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:38:33.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.826+0000 7f35367fc640 1 -- 172.21.15.107:0/3639837683 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3520003520 con 0x7f35381430c0 2024-03-25T19:38:33.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.832+0000 7f35367fc640 1 -- 172.21.15.107:0/3639837683 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537680469 0 0) 0x7f3538142520 con 0x7f35380a98f0 2024-03-25T19:38:33.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.832+0000 7f35367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:33.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.832+0000 7f35367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:33.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.832+0000 7f35367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:33.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.832+0000 7f35367fc640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:38:33.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.832+0000 7f35367fc640 1 -- 172.21.15.107:0/3639837683 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35380b2a30 con 0x7f35380a98f0 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 1 -- 172.21.15.107:0/3639837683 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3793356777 0 0) 0x7f3520003520 con 0x7f35381430c0 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 1 -- 172.21.15.107:0/3639837683 >> v1:172.21.15.195:6789/0 conn(0x7f353813f710 legacy=0x7f35381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 1 -- 172.21.15.107:0/3639837683 >> v1:172.21.15.107:6789/0 conn(0x7f35380a98f0 legacy=0x7f35380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 1 monclient: found mon.noname-b 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 1 -- 172.21.15.107:0/3639837683 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35380b44c0 con 0x7f35381430c0 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 1 -- 172.21.15.107:0/3639837683 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3530001a10 con 0x7f35381430c0 2024-03-25T19:38:33.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f353e120640 5 monclient: authenticate success, global_id 4346 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f353e120640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f35367fc640 10 monclient: dump: 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:33.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.840+0000 7f353e120640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.852+0000 7f35367fc640 1 -- 172.21.15.107:0/3639837683 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3530001b70 con 0x7f35381430c0 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.852+0000 7f35367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.852+0000 7f35367fc640 1 -- 172.21.15.107:0/3639837683 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3530001f80 con 0x7f35381430c0 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.852+0000 7f35367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.852+0000 7f35367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.852+0000 7f35367fc640 10 monclient: dump: 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:33.852 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:33.853 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:33.853 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:33.853 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:33.853 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:33.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.852+0000 7f353e120640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:33.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.852+0000 7f353e120640 4 set_mon_vals no callback set 2024-03-25T19:38:33.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.853+0000 7f353e120640 10 monclient: shutdown 2024-03-25T19:38:33.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.853+0000 7f353e120640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:33.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.853+0000 7f353e120640 1 -- 172.21.15.107:0/3639837683 >> v1:172.21.15.107:6790/0 conn(0x7f35381430c0 legacy=0x7f353814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:33.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.853+0000 7f353e120640 1 -- 172.21.15.107:0/3639837683 shutdown_connections 2024-03-25T19:38:33.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.853+0000 7f353e120640 1 -- 172.21.15.107:0/3639837683 >> 172.21.15.107:0/3639837683 conn(0x7f35380b7ea0 msgr2=0x7f35380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:33.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.853+0000 7f353e120640 1 -- 172.21.15.107:0/3639837683 shutdown_connections 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.853+0000 7f353e120640 1 -- 172.21.15.107:0/3639837683 wait complete. 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.854+0000 7f353e120640 10 monclient: build_initial_monmap 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.854+0000 7f353e120640 10 monclient: monmap: 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.854+0000 7f353e120640 1 librados: starting msgr at 2024-03-25T19:38:33.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.854+0000 7f353e120640 1 librados: starting objecter 2024-03-25T19:38:33.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.854+0000 7f353e120640 1 Processor -- start 2024-03-25T19:38:33.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 1 -- start start 2024-03-25T19:38:33.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 1 librados: setting wanted keys 2024-03-25T19:38:33.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 1 librados: calling monclient init 2024-03-25T19:38:33.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 10 monclient: init 2024-03-25T19:38:33.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:33.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:33.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 10 monclient(hunting): picked mon.noname-a con 0x7f35380a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:33.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 10 monclient(hunting): picked mon.noname-b con 0x7f353813f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:33.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 10 monclient(hunting): picked mon.noname-c con 0x7f35381430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:33.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35381daca0 con 0x7f35380a98f0 2024-03-25T19:38:33.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3538138ab0 con 0x7f353813f710 2024-03-25T19:38:33.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35380aba90 con 0x7f35381430c0 2024-03-25T19:38:33.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:33.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.855+0000 7f353e120640 10 monclient(hunting): authenticate will time out at 4706.196573s 2024-03-25T19:38:33.872 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.873+0000 7f3537fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f35381430c0 0x7f35381d47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44658/0 (socket says 172.21.15.107:44658) 2024-03-25T19:38:33.872 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.873+0000 7f3537fff640 1 -- 172.21.15.107:0/1022943393 learned_addr learned my addr 172.21.15.107:0/1022943393 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:33.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.875+0000 7f351ffff640 10 client.?.objecter ms_handle_connect 0x7f35381430c0 2024-03-25T19:38:33.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.875+0000 7f351ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:33.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.879+0000 7f351ffff640 10 client.?.objecter ms_handle_connect 0x7f353813f710 2024-03-25T19:38:33.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.879+0000 7f351ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:33.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.886+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182344413 0 0) 0x7f3538138ab0 con 0x7f353813f710 2024-03-25T19:38:33.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.886+0000 7f351ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:33.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.886+0000 7f351ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:33.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.886+0000 7f351ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:33.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.886+0000 7f351ffff640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:38:33.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.886+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3524004eb0 con 0x7f353813f710 2024-03-25T19:38:33.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.887+0000 7f351ffff640 10 client.?.objecter ms_handle_connect 0x7f35380a98f0 2024-03-25T19:38:33.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.887+0000 7f351ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:33.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.900+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668397867 0 0) 0x7f35380aba90 con 0x7f35381430c0 2024-03-25T19:38:33.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.900+0000 7f351ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:33.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.900+0000 7f351ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:33.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.900+0000 7f351ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:33.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.900+0000 7f351ffff640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:38:33.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.900+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3538138ab0 con 0x7f35381430c0 2024-03-25T19:38:33.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.906+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089885467 0 0) 0x7f35381daca0 con 0x7f35380a98f0 2024-03-25T19:38:33.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.906+0000 7f351ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:33.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.906+0000 7f351ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:33.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.906+0000 7f351ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:33.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.906+0000 7f351ffff640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:38:33.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.906+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35380aba90 con 0x7f35380a98f0 2024-03-25T19:38:33.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.912+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 35960545 0 0) 0x7f3524004eb0 con 0x7f353813f710 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.912+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35381daca0 con 0x7f353813f710 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.912+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f352c001930 con 0x7f353813f710 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.912+0000 7f351ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.912+0000 7f351ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.912+0000 7f351ffff640 10 monclient(hunting): dump: 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:33.912 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:33.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.912+0000 7f351ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:33.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.919+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3189882281 0 0) 0x7f3538138ab0 con 0x7f35381430c0 2024-03-25T19:38:33.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.919+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3524004eb0 con 0x7f35381430c0 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.919+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3530001e80 con 0x7f35381430c0 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.919+0000 7f351ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.919+0000 7f351ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.919+0000 7f351ffff640 10 monclient(hunting): dump: 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:33.919 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:33.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.924+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2225261866 0 0) 0x7f35380aba90 con 0x7f35380a98f0 2024-03-25T19:38:33.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.924+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3538138ab0 con 0x7f35380a98f0 2024-03-25T19:38:33.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.924+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3528001930 con 0x7f35380a98f0 2024-03-25T19:38:33.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.924+0000 7f351ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:33.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.924+0000 7f351ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:33.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.924+0000 7f351ffff640 10 monclient(hunting): dump: 2024-03-25T19:38:33.923 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:33.924 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:33.924 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:33.924 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:33.924 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:33.924 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:33.924 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:33.924 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:33.924 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:33.924 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1321887676 0 0) 0x7f35381daca0 con 0x7f353813f710 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 >> v1:172.21.15.107:6790/0 conn(0x7f35381430c0 legacy=0x7f35381d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 >> v1:172.21.15.107:6789/0 conn(0x7f35380a98f0 legacy=0x7f35381383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f351ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f351ffff640 1 monclient: found mon.b 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f351ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f351ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35381d41e0 con 0x7f353813f710 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f351ffff640 10 monclient: _finish_auth 0 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f351ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f351ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f353e120640 5 monclient: authenticate success, global_id 4336 2024-03-25T19:38:33.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f353e120640 10 monclient: _renew_subs 2024-03-25T19:38:33.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f353e120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:38:33.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f353e120640 1 -- 172.21.15.107:0/1022943393 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35381db840 con 0x7f353813f710 2024-03-25T19:38:33.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f353e120640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:33.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f353e120640 10 monclient: _renew_subs 2024-03-25T19:38:33.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f353e120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:38:33.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f353e120640 1 -- 172.21.15.107:0/1022943393 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3538138f00 con 0x7f353813f710 2024-03-25T19:38:33.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.928+0000 7f353e120640 1 librados: init done 2024-03-25T19:38:33.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.929+0000 7f353e120640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:33.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.929+0000 7f353e120640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:33.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.929+0000 7f353e120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:38:33.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:33.929+0000 7f353e120640 1 -- 172.21.15.107:0/1022943393 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35381dae50 con 0x7f353813f710 2024-03-25T19:38:34.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.206+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f352c0035f0 con 0x7f353813f710 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.206+0000 7f351ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.206+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f352c003760 con 0x7f353813f710 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.206+0000 7f351ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.206+0000 7f351ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.206+0000 7f351ffff640 10 monclient: dump: 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:34.206 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:34.207 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:34.207 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:34.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.206+0000 7f3535ffb640 4 set_mon_vals no callback set 2024-03-25T19:38:34.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.206+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f352c019730 con 0x7f353813f710 2024-03-25T19:38:34.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.207+0000 7f35377fe640 10 monclient: get_auth_request con 0x7f352406e0b0 auth_method 0 2024-03-25T19:38:34.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.207+0000 7f351ffff640 10 client.4336.objecter ms_handle_connect 0x7f352406e0b0 2024-03-25T19:38:34.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.211+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f352c053450 con 0x7f353813f710 2024-03-25T19:38:34.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.211+0000 7f351ffff640 10 client.4336.objecter ms_dispatch 0x7f3538000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:34.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.211+0000 7f351ffff640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:34.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.211+0000 7f351ffff640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:34.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.211+0000 7f351ffff640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T19:38:34.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.211+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f352c04ed90 con 0x7f353813f710 2024-03-25T19:38:34.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.211+0000 7f351ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:34.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.211+0000 7f351ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:34.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.434+0000 7f353e120640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:34.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.434+0000 7f353e120640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:34.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.434+0000 7f353e120640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:38:34.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.434+0000 7f353e120640 1 -- 172.21.15.107:0/1022943393 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35380627b0 con 0x7f353813f710 2024-03-25T19:38:34.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.473+0000 7f351ffff640 1 -- 172.21.15.107:0/1022943393 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f352c050e90 con 0x7f353813f710 2024-03-25T19:38:34.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.473+0000 7f351ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:34.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.473+0000 7f351ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:34.473 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:38:34.473 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 10 librados: watch_flush enter 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 10 librados: watch_flush exit 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 10 client.4336.objecter successfully canceled tick 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 1 -- 172.21.15.107:0/1022943393 >> v1:172.21.15.195:6800/2984408421 conn(0x7f352406e0b0 legacy=0x7f3524070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 10 monclient: shutdown 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 1 -- 172.21.15.107:0/1022943393 >> v1:172.21.15.195:6789/0 conn(0x7f353813f710 legacy=0x7f35381db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 1 -- 172.21.15.107:0/1022943393 shutdown_connections 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 1 -- 172.21.15.107:0/1022943393 >> 172.21.15.107:0/1022943393 conn(0x7f35380b7ea0 msgr2=0x7f3538074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 1 -- 172.21.15.107:0/1022943393 shutdown_connections 2024-03-25T19:38:34.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 1 -- 172.21.15.107:0/1022943393 wait complete. 2024-03-25T19:38:34.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:34.476+0000 7f353e120640 1 librados: shutdown 2024-03-25T19:38:34.487 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:41.292 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:41.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.399+0000 7f3068bf4640 10 monclient: get_monmap_and_config 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.399+0000 7f3068bf4640 10 monclient: build_initial_monmap 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.399+0000 7f3068bf4640 10 monclient: monmap: 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:38:41.400471+0000 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:38:41.400471+0000 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:38:41.399 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:41.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.400+0000 7f3068bf4640 1 Processor -- start 2024-03-25T19:38:41.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.400+0000 7f3068bf4640 1 -- start start 2024-03-25T19:38:41.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.400+0000 7f3068bf4640 10 monclient: init 2024-03-25T19:38:41.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.401+0000 7f3068bf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:41.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.401+0000 7f3068bf4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:41.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.401+0000 7f3068bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f30640705b0 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:41.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.401+0000 7f3068bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f30641402d0 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:41.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.401+0000 7f3068bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f3064149ae0 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:41.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.401+0000 7f3068bf4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30640718c0 con 0x7f30641402d0 2024-03-25T19:38:41.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.401+0000 7f3068bf4640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30640714c0 con 0x7f30640705b0 2024-03-25T19:38:41.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.401+0000 7f3068bf4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3064072b50 con 0x7f3064149ae0 2024-03-25T19:38:41.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.401+0000 7f3068bf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:41.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.401+0000 7f3068bf4640 10 monclient(hunting): authenticate will time out at 4713.742356s 2024-03-25T19:38:41.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.402+0000 7f306259c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f30640705b0 0x7f306413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:59610/0 (socket says 172.21.15.107:59610) 2024-03-25T19:38:41.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.402+0000 7f306259c640 1 -- 172.21.15.107:0/3618130446 learned_addr learned my addr 172.21.15.107:0/3618130446 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:41.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.410+0000 7f306159a640 1 -- 172.21.15.107:0/3618130446 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023493108 0 0) 0x7f30640714c0 con 0x7f30640705b0 2024-03-25T19:38:41.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.410+0000 7f306159a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:41.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.410+0000 7f306159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:41.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.410+0000 7f306159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:41.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.410+0000 7f306159a640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:38:41.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.410+0000 7f306159a640 1 -- 172.21.15.107:0/3618130446 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f304c003520 con 0x7f30640705b0 2024-03-25T19:38:41.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 1 -- 172.21.15.107:0/3618130446 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500221663 0 0) 0x7f30640718c0 con 0x7f30641402d0 2024-03-25T19:38:41.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:41.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 1 -- 172.21.15.107:0/3618130446 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30640714c0 con 0x7f30641402d0 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 1 -- 172.21.15.107:0/3618130446 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3078828264 0 0) 0x7f304c003520 con 0x7f30640705b0 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 1 -- 172.21.15.107:0/3618130446 >> v1:172.21.15.107:6790/0 conn(0x7f3064149ae0 legacy=0x7f306414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 1 -- 172.21.15.107:0/3618130446 >> v1:172.21.15.107:6789/0 conn(0x7f30641402d0 legacy=0x7f3064140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 10 monclient: _finish_hunting 0 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 1 monclient: found mon.noname-c 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 1 -- 172.21.15.107:0/3618130446 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306413dd10 con 0x7f30640705b0 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 10 monclient: _finish_auth 0 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 10 monclient: _check_auth_tickets 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 1 -- 172.21.15.107:0/3618130446 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3054001690 con 0x7f30640705b0 2024-03-25T19:38:41.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f3068bf4640 5 monclient: authenticate success, global_id 4339 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f3068bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f306159a640 10 monclient: dump: 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:41.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.416+0000 7f3068bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:41.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.431+0000 7f306159a640 1 -- 172.21.15.107:0/3618130446 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3054001870 con 0x7f30640705b0 2024-03-25T19:38:41.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.432+0000 7f306159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:41.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.432+0000 7f3068bf4640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:41.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.432+0000 7f3068bf4640 4 set_mon_vals no callback set 2024-03-25T19:38:41.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.432+0000 7f3068bf4640 10 monclient: shutdown 2024-03-25T19:38:41.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.432+0000 7f3068bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:41.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.432+0000 7f3068bf4640 1 -- 172.21.15.107:0/3618130446 >> v1:172.21.15.195:6789/0 conn(0x7f30640705b0 legacy=0x7f306413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:41.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.432+0000 7f3068bf4640 1 -- 172.21.15.107:0/3618130446 shutdown_connections 2024-03-25T19:38:41.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.432+0000 7f3068bf4640 1 -- 172.21.15.107:0/3618130446 >> 172.21.15.107:0/3618130446 conn(0x7f30640b5d10 msgr2=0x7f30640b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:41.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.433+0000 7f3068bf4640 1 -- 172.21.15.107:0/3618130446 shutdown_connections 2024-03-25T19:38:41.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.433+0000 7f3068bf4640 1 -- 172.21.15.107:0/3618130446 wait complete. 2024-03-25T19:38:41.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.433+0000 7f3068bf4640 10 monclient: build_initial_monmap 2024-03-25T19:38:41.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.433+0000 7f3068bf4640 10 monclient: monmap: 2024-03-25T19:38:41.433 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:41.433 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:41.433 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:38:41.433 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:38:41.433 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:41.433 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:38:41.433 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:41.433 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:38:41.433 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:38:41.434 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:41.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.434+0000 7f3068bf4640 1 librados: starting msgr at 2024-03-25T19:38:41.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.434+0000 7f3068bf4640 1 librados: starting objecter 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.434+0000 7f3068bf4640 1 Processor -- start 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.434+0000 7f3068bf4640 1 -- start start 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.434+0000 7f3068bf4640 1 librados: setting wanted keys 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.434+0000 7f3068bf4640 1 librados: calling monclient init 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.434+0000 7f3068bf4640 10 monclient: init 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.435+0000 7f3068bf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.435+0000 7f3068bf4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.435+0000 7f3068bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f30640705b0 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.435+0000 7f3068bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f30641402d0 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.435+0000 7f3068bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f3064149ae0 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.435+0000 7f3068bf4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30641daab0 con 0x7f30641402d0 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.435+0000 7f3068bf4640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30641db6e0 con 0x7f30640705b0 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.435+0000 7f3068bf4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30641db870 con 0x7f3064149ae0 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.435+0000 7f3068bf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:41.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.435+0000 7f3068bf4640 10 monclient(hunting): authenticate will time out at 4713.775928s 2024-03-25T19:38:41.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.439+0000 7f3062d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3064149ae0 0x7f3064139e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55408/0 (socket says 172.21.15.107:55408) 2024-03-25T19:38:41.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.439+0000 7f3062d9d640 1 -- 172.21.15.107:0/4042162578 learned_addr learned my addr 172.21.15.107:0/4042162578 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:41.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.444+0000 7f3052ffd640 10 client.?.objecter ms_handle_connect 0x7f3064149ae0 2024-03-25T19:38:41.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.444+0000 7f3052ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:41.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.452+0000 7f3052ffd640 10 client.?.objecter ms_handle_connect 0x7f30640705b0 2024-03-25T19:38:41.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.452+0000 7f3052ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:41.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.457+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551446828 0 0) 0x7f30641db6e0 con 0x7f30640705b0 2024-03-25T19:38:41.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.457+0000 7f3052ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:41.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.457+0000 7f3052ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:41.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.457+0000 7f3052ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:41.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.457+0000 7f3052ffd640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:38:41.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.457+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3044004eb0 con 0x7f30640705b0 2024-03-25T19:38:41.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.461+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929036011 0 0) 0x7f30641db870 con 0x7f3064149ae0 2024-03-25T19:38:41.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.461+0000 7f3052ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:41.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.461+0000 7f3052ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:41.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.461+0000 7f3052ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:41.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.461+0000 7f3052ffd640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:38:41.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.461+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30641db6e0 con 0x7f3064149ae0 2024-03-25T19:38:41.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.464+0000 7f3052ffd640 10 client.?.objecter ms_handle_connect 0x7f30641402d0 2024-03-25T19:38:41.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.464+0000 7f3052ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:41.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.484+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1846721477 0 0) 0x7f3044004eb0 con 0x7f30640705b0 2024-03-25T19:38:41.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.484+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30641db870 con 0x7f30640705b0 2024-03-25T19:38:41.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.484+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3054001650 con 0x7f30640705b0 2024-03-25T19:38:41.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.484+0000 7f3052ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.484+0000 7f3052ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.484+0000 7f3052ffd640 10 monclient(hunting): dump: 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:41.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.484+0000 7f3052ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:41.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.485+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943337654 0 0) 0x7f30641daab0 con 0x7f30641402d0 2024-03-25T19:38:41.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.485+0000 7f3052ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:41.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.485+0000 7f3052ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:41.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.485+0000 7f3052ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:41.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.485+0000 7f3052ffd640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:38:41.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.485+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3044004eb0 con 0x7f30641402d0 2024-03-25T19:38:41.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.489+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4156283785 0 0) 0x7f30641db6e0 con 0x7f3064149ae0 2024-03-25T19:38:41.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.489+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30641daab0 con 0x7f3064149ae0 2024-03-25T19:38:41.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.489+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f305c001b90 con 0x7f3064149ae0 2024-03-25T19:38:41.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.489+0000 7f3052ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:41.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.489+0000 7f3052ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:41.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.489+0000 7f3052ffd640 10 monclient(hunting): dump: 2024-03-25T19:38:41.489 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:41.489 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:41.489 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:41.490 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:41.490 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:41.490 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:41.490 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:41.490 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:41.490 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:41.490 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3685359314 0 0) 0x7f30641daab0 con 0x7f3064149ae0 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 >> v1:172.21.15.195:6789/0 conn(0x7f30640705b0 legacy=0x7f30641dafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 >> v1:172.21.15.107:6789/0 conn(0x7f30641402d0 legacy=0x7f30640aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3052ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3052ffd640 1 monclient: found mon.c 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3052ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3052ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30640aaac0 con 0x7f3064149ae0 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3052ffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3052ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3068bf4640 5 monclient: authenticate success, global_id 4355 2024-03-25T19:38:41.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3068bf4640 10 monclient: _renew_subs 2024-03-25T19:38:41.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3068bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:41.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3068bf4640 1 -- 172.21.15.107:0/4042162578 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3064141f40 con 0x7f3064149ae0 2024-03-25T19:38:41.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3068bf4640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:41.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3068bf4640 10 monclient: _renew_subs 2024-03-25T19:38:41.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3068bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:41.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3068bf4640 1 -- 172.21.15.107:0/4042162578 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30640a9910 con 0x7f3064149ae0 2024-03-25T19:38:41.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.501+0000 7f3068bf4640 1 librados: init done 2024-03-25T19:38:41.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.502+0000 7f3068bf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:41.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.502+0000 7f3068bf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:41.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.502+0000 7f3068bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:41.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.502+0000 7f3068bf4640 1 -- 172.21.15.107:0/4042162578 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30640abc00 con 0x7f3064149ae0 2024-03-25T19:38:41.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.810+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f305c002080 con 0x7f3064149ae0 2024-03-25T19:38:41.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.810+0000 7f3052ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:41.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.810+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f305c003b30 con 0x7f3064149ae0 2024-03-25T19:38:41.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.810+0000 7f3052ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:41.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.810+0000 7f3052ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:41.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.810+0000 7f3052ffd640 10 monclient: dump: 2024-03-25T19:38:41.810 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:41.810 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:41.811 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:41.811 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:41.811 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:41.811 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:41.811 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:41.811 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:41.811 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:41.811 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:41.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.810+0000 7f3060d99640 4 set_mon_vals no callback set 2024-03-25T19:38:41.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.811+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f305c003d50 con 0x7f3064149ae0 2024-03-25T19:38:41.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.811+0000 7f306259c640 10 monclient: get_auth_request con 0x7f304406df50 auth_method 0 2024-03-25T19:38:41.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.812+0000 7f3052ffd640 10 client.4355.objecter ms_handle_connect 0x7f304406df50 2024-03-25T19:38:41.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.819+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f305c054990 con 0x7f3064149ae0 2024-03-25T19:38:41.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.819+0000 7f3052ffd640 10 client.4355.objecter ms_dispatch 0x7f3064000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:41.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.819+0000 7f3052ffd640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:41.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.819+0000 7f3052ffd640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:41.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.819+0000 7f3052ffd640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-25T19:38:41.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.819+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f305c0504a0 con 0x7f3064149ae0 2024-03-25T19:38:41.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.819+0000 7f3052ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:41.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:41.819+0000 7f3052ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:42.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.064+0000 7f3068bf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:42.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.064+0000 7f3068bf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:42.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.064+0000 7f3068bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:42.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.064+0000 7f3068bf4640 1 -- 172.21.15.107:0/4042162578 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30641da910 con 0x7f3064149ae0 2024-03-25T19:38:42.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.090+0000 7f3052ffd640 1 -- 172.21.15.107:0/4042162578 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f30641da910 con 0x7f3064149ae0 2024-03-25T19:38:42.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.090+0000 7f3052ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:42.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.090+0000 7f3052ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:42.090 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:38:42.090 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.092+0000 7f3068bf4640 10 librados: watch_flush enter 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.092+0000 7f3068bf4640 10 librados: watch_flush exit 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.092+0000 7f3068bf4640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.092+0000 7f3068bf4640 10 client.4355.objecter successfully canceled tick 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.092+0000 7f3068bf4640 1 -- 172.21.15.107:0/4042162578 >> v1:172.21.15.195:6800/2984408421 conn(0x7f304406df50 legacy=0x7f3044070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.092+0000 7f3068bf4640 10 monclient: shutdown 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.093+0000 7f3068bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.093+0000 7f3068bf4640 1 -- 172.21.15.107:0/4042162578 >> v1:172.21.15.107:6790/0 conn(0x7f3064149ae0 legacy=0x7f3064139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.093+0000 7f3068bf4640 1 -- 172.21.15.107:0/4042162578 shutdown_connections 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.093+0000 7f3068bf4640 1 -- 172.21.15.107:0/4042162578 >> 172.21.15.107:0/4042162578 conn(0x7f30640b5d10 msgr2=0x7f30640b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.093+0000 7f3068bf4640 1 -- 172.21.15.107:0/4042162578 shutdown_connections 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.093+0000 7f3068bf4640 1 -- 172.21.15.107:0/4042162578 wait complete. 2024-03-25T19:38:42.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:42.093+0000 7f3068bf4640 1 librados: shutdown 2024-03-25T19:38:42.104 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:44.721 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:03:15.037 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:44.773 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:03:15.090 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:44.794 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:03:15.110 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:44.818 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:03:15.105 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:44.835 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:03:15.151 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:44.840 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:03:15.127 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:44.852 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:03:15.169 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:44.866 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:03:15.153 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:44.881 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:03:15.198 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:48.912 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:49.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.028+0000 7fb3b7367640 10 monclient: get_monmap_and_config 2024-03-25T19:38:49.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.028+0000 7fb3b7367640 10 monclient: build_initial_monmap 2024-03-25T19:38:49.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.028+0000 7fb3b7367640 10 monclient: monmap: 2024-03-25T19:38:49.028 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:49.029 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:49.029 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:38:49.029900+0000 2024-03-25T19:38:49.029 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:38:49.029900+0000 2024-03-25T19:38:49.029 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:49.029 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:49.029 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:49.029 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:38:49.029 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:38:49.029 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:49.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.029+0000 7fb3b7367640 1 Processor -- start 2024-03-25T19:38:49.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.030+0000 7fb3b7367640 1 -- start start 2024-03-25T19:38:49.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.030+0000 7fb3b7367640 10 monclient: init 2024-03-25T19:38:49.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.030+0000 7fb3b7367640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:49.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.030+0000 7fb3b7367640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:49.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.030+0000 7fb3b7367640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b00a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:49.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.031+0000 7fb3b7367640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b013f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:49.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.031+0000 7fb3b7367640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b01430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:49.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.031+0000 7fb3b7367640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b0142520 con 0x7fb3b01430c0 2024-03-25T19:38:49.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.031+0000 7fb3b7367640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b00b28a0 con 0x7fb3b00a98f0 2024-03-25T19:38:49.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.031+0000 7fb3b7367640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b00b2a30 con 0x7fb3b013f710 2024-03-25T19:38:49.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.031+0000 7fb3b7367640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:49.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.031+0000 7fb3b7367640 10 monclient(hunting): authenticate will time out at 4721.371814s 2024-03-25T19:38:49.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.041+0000 7fb3b5103640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb3b00a98f0 0x7fb3b00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:36298/0 (socket says 172.21.15.107:36298) 2024-03-25T19:38:49.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.041+0000 7fb3b5103640 1 -- 172.21.15.107:0/2491733096 learned_addr learned my addr 172.21.15.107:0/2491733096 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:49.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.054+0000 7fb3a7fff640 1 -- 172.21.15.107:0/2491733096 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259426116 0 0) 0x7fb3b00b28a0 con 0x7fb3b00a98f0 2024-03-25T19:38:49.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.054+0000 7fb3a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:49.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.054+0000 7fb3a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:49.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.054+0000 7fb3a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:49.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.054+0000 7fb3a7fff640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:38:49.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.054+0000 7fb3a7fff640 1 -- 172.21.15.107:0/2491733096 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb39c003520 con 0x7fb3b00a98f0 2024-03-25T19:38:49.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.060+0000 7fb3a7fff640 1 -- 172.21.15.107:0/2491733096 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100835295 0 0) 0x7fb3b00b2a30 con 0x7fb3b013f710 2024-03-25T19:38:49.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.060+0000 7fb3a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:49.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.060+0000 7fb3a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:49.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.060+0000 7fb3a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:49.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.060+0000 7fb3a7fff640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:38:49.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.060+0000 7fb3a7fff640 1 -- 172.21.15.107:0/2491733096 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b00b28a0 con 0x7fb3b013f710 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 1 -- 172.21.15.107:0/2491733096 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2225486566 0 0) 0x7fb39c003520 con 0x7fb3b00a98f0 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 1 -- 172.21.15.107:0/2491733096 >> v1:172.21.15.107:6790/0 conn(0x7fb3b013f710 legacy=0x7fb3b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 1 -- 172.21.15.107:0/2491733096 >> v1:172.21.15.107:6789/0 conn(0x7fb3b01430c0 legacy=0x7fb3b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 1 monclient: found mon.noname-c 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 1 -- 172.21.15.107:0/2491733096 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b00b44c0 con 0x7fb3b00a98f0 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 10 monclient: _finish_auth 0 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 1 -- 172.21.15.107:0/2491733096 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb3a8001a10 con 0x7fb3b00a98f0 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3a7fff640 10 monclient: dump: 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:49.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3b7367640 5 monclient: authenticate success, global_id 4345 2024-03-25T19:38:49.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.069+0000 7fb3b7367640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:49.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.075+0000 7fb3a7fff640 1 -- 172.21.15.107:0/2491733096 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3a8001b70 con 0x7fb3b00a98f0 2024-03-25T19:38:49.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.075+0000 7fb3a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:49.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.075+0000 7fb3a7fff640 1 -- 172.21.15.107:0/2491733096 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb3a8001f80 con 0x7fb3b00a98f0 2024-03-25T19:38:49.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.075+0000 7fb3a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:49.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.075+0000 7fb3a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:49.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.075+0000 7fb3a7fff640 10 monclient: dump: 2024-03-25T19:38:49.075 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:49.075 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:49.075 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:49.075 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:49.076 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:49.076 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:49.076 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:49.076 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:49.076 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:49.076 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:49.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.075+0000 7fb3b7367640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:49.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.075+0000 7fb3b7367640 4 set_mon_vals no callback set 2024-03-25T19:38:49.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.076+0000 7fb3b7367640 10 monclient: shutdown 2024-03-25T19:38:49.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.076+0000 7fb3b7367640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:49.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.076+0000 7fb3b7367640 1 -- 172.21.15.107:0/2491733096 >> v1:172.21.15.195:6789/0 conn(0x7fb3b00a98f0 legacy=0x7fb3b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:49.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.076+0000 7fb3b7367640 1 -- 172.21.15.107:0/2491733096 shutdown_connections 2024-03-25T19:38:49.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.076+0000 7fb3b7367640 1 -- 172.21.15.107:0/2491733096 >> 172.21.15.107:0/2491733096 conn(0x7fb3b00b7ea0 msgr2=0x7fb3b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:49.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.076+0000 7fb3b7367640 1 -- 172.21.15.107:0/2491733096 shutdown_connections 2024-03-25T19:38:49.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.076+0000 7fb3b7367640 1 -- 172.21.15.107:0/2491733096 wait complete. 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.077+0000 7fb3b7367640 10 monclient: build_initial_monmap 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.077+0000 7fb3b7367640 10 monclient: monmap: 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.077+0000 7fb3b7367640 1 librados: starting msgr at 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.077+0000 7fb3b7367640 1 librados: starting objecter 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.077+0000 7fb3b7367640 1 Processor -- start 2024-03-25T19:38:49.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.077+0000 7fb3b7367640 1 -- start start 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.077+0000 7fb3b7367640 1 librados: setting wanted keys 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.077+0000 7fb3b7367640 1 librados: calling monclient init 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.077+0000 7fb3b7367640 10 monclient: init 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.078+0000 7fb3b7367640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.078+0000 7fb3b7367640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.078+0000 7fb3b7367640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b00a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.078+0000 7fb3b7367640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b013f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.078+0000 7fb3b7367640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b01430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.078+0000 7fb3b7367640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b0135730 con 0x7fb3b013f710 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.078+0000 7fb3b7367640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b01358c0 con 0x7fb3b01430c0 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.078+0000 7fb3b7367640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b0135a50 con 0x7fb3b00a98f0 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.078+0000 7fb3b7367640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:49.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.078+0000 7fb3b7367640 10 monclient(hunting): authenticate will time out at 4721.419332s 2024-03-25T19:38:49.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.089+0000 7fb3b4902640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb3b013f710 0x7fb3b01d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42768/0 (socket says 172.21.15.107:42768) 2024-03-25T19:38:49.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.089+0000 7fb3b4902640 1 -- 172.21.15.107:0/2777717972 learned_addr learned my addr 172.21.15.107:0/2777717972 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:49.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.089+0000 7fb3a5ffb640 10 client.?.objecter ms_handle_connect 0x7fb3b013f710 2024-03-25T19:38:49.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.090+0000 7fb3a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:49.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.090+0000 7fb3a5ffb640 10 client.?.objecter ms_handle_connect 0x7fb3b00a98f0 2024-03-25T19:38:49.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.090+0000 7fb3a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.093+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325822128 0 0) 0x7fb3b0135a50 con 0x7fb3b00a98f0 2024-03-25T19:38:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.093+0000 7fb3a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.093+0000 7fb3a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.093+0000 7fb3a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.093+0000 7fb3a5ffb640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:38:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.093+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb394004eb0 con 0x7fb3b00a98f0 2024-03-25T19:38:49.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.095+0000 7fb3a5ffb640 10 client.?.objecter ms_handle_connect 0x7fb3b01430c0 2024-03-25T19:38:49.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.095+0000 7fb3a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:49.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.101+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157457679 0 0) 0x7fb3b0135730 con 0x7fb3b013f710 2024-03-25T19:38:49.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.101+0000 7fb3a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:49.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.101+0000 7fb3a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:49.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.101+0000 7fb3a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:49.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.101+0000 7fb3a5ffb640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:38:49.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.101+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b0135a50 con 0x7fb3b013f710 2024-03-25T19:38:49.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.109+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2103816309 0 0) 0x7fb394004eb0 con 0x7fb3b00a98f0 2024-03-25T19:38:49.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.109+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3b0135730 con 0x7fb3b00a98f0 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.109+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb3a8001e80 con 0x7fb3b00a98f0 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.109+0000 7fb3a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.109+0000 7fb3a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.109+0000 7fb3a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:49.109 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:49.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.109+0000 7fb3a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:49.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827003832 0 0) 0x7fb3b01358c0 con 0x7fb3b01430c0 2024-03-25T19:38:49.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:49.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb394004eb0 con 0x7fb3b01430c0 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1300558382 0 0) 0x7fb3b0135a50 con 0x7fb3b013f710 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3b01358c0 con 0x7fb3b013f710 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb3a00019f0 con 0x7fb3b013f710 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.110+0000 7fb3a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:49.111 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:49.112 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:49.112 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:49.112 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:49.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3915976286 0 0) 0x7fb3b0135730 con 0x7fb3b00a98f0 2024-03-25T19:38:49.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 >> v1:172.21.15.195:6789/0 conn(0x7fb3b01430c0 legacy=0x7fb3b01d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:49.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 >> v1:172.21.15.107:6789/0 conn(0x7fb3b013f710 legacy=0x7fb3b01d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:49.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:49.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3a5ffb640 1 monclient: found mon.c 2024-03-25T19:38:49.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:49.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b01d5000 con 0x7fb3b00a98f0 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3a5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3b7367640 5 monclient: authenticate success, global_id 4361 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3b7367640 10 monclient: _renew_subs 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3b7367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3b7367640 1 -- 172.21.15.107:0/2777717972 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b01d52b0 con 0x7fb3b00a98f0 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3b7367640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3b7367640 10 monclient: _renew_subs 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3b7367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:49.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3b7367640 1 -- 172.21.15.107:0/2777717972 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b013c900 con 0x7fb3b00a98f0 2024-03-25T19:38:49.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.121+0000 7fb3b7367640 1 librados: init done 2024-03-25T19:38:49.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.122+0000 7fb3b7367640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:49.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.122+0000 7fb3b7367640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:49.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.122+0000 7fb3b7367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:49.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.122+0000 7fb3b7367640 1 -- 172.21.15.107:0/2777717972 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b013cbb0 con 0x7fb3b00a98f0 2024-03-25T19:38:49.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.381+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3a80015a0 con 0x7fb3b00a98f0 2024-03-25T19:38:49.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.381+0000 7fb3a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:49.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.381+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb3a8004e10 con 0x7fb3b00a98f0 2024-03-25T19:38:49.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.381+0000 7fb3a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:49.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.381+0000 7fb3a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.381+0000 7fb3a5ffb640 10 monclient: dump: 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:49.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.381+0000 7fb3a77fe640 4 set_mon_vals no callback set 2024-03-25T19:38:49.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.382+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fb3a801ae20 con 0x7fb3b00a98f0 2024-03-25T19:38:49.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.383+0000 7fb3b4902640 10 monclient: get_auth_request con 0x7fb39406dfb0 auth_method 0 2024-03-25T19:38:49.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.383+0000 7fb3a5ffb640 10 client.4361.objecter ms_handle_connect 0x7fb39406dfb0 2024-03-25T19:38:49.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.387+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fb3a8054a90 con 0x7fb3b00a98f0 2024-03-25T19:38:49.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.387+0000 7fb3a5ffb640 10 client.4361.objecter ms_dispatch 0x7fb3b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:49.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.387+0000 7fb3a5ffb640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:49.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.387+0000 7fb3a5ffb640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:49.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.387+0000 7fb3a5ffb640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-25T19:38:49.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.387+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3a80503d0 con 0x7fb3b00a98f0 2024-03-25T19:38:49.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.387+0000 7fb3a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:49.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.387+0000 7fb3a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:49.652 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.652+0000 7fb3b7367640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:49.652 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.652+0000 7fb3b7367640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:49.652 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.652+0000 7fb3b7367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:38:49.652 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.652+0000 7fb3b7367640 1 -- 172.21.15.107:0/2777717972 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3b01371c0 con 0x7fb3b00a98f0 2024-03-25T19:38:49.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.676+0000 7fb3a5ffb640 1 -- 172.21.15.107:0/2777717972 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fb3a80524d0 con 0x7fb3b00a98f0 2024-03-25T19:38:49.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.676+0000 7fb3a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:49.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.676+0000 7fb3a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:49.676 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:38:49.676 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:49.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 10 librados: watch_flush enter 2024-03-25T19:38:49.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 10 librados: watch_flush exit 2024-03-25T19:38:49.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-25T19:38:49.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 10 client.4361.objecter successfully canceled tick 2024-03-25T19:38:49.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 1 -- 172.21.15.107:0/2777717972 >> v1:172.21.15.195:6800/2984408421 conn(0x7fb39406dfb0 legacy=0x7fb394070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:49.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 10 monclient: shutdown 2024-03-25T19:38:49.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:49.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 1 -- 172.21.15.107:0/2777717972 >> v1:172.21.15.107:6790/0 conn(0x7fb3b00a98f0 legacy=0x7fb3b01dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:49.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 1 -- 172.21.15.107:0/2777717972 shutdown_connections 2024-03-25T19:38:49.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 1 -- 172.21.15.107:0/2777717972 >> 172.21.15.107:0/2777717972 conn(0x7fb3b00b7ea0 msgr2=0x7fb3b014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:49.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 1 -- 172.21.15.107:0/2777717972 shutdown_connections 2024-03-25T19:38:49.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 1 -- 172.21.15.107:0/2777717972 wait complete. 2024-03-25T19:38:49.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:49.679+0000 7fb3b7367640 1 librados: shutdown 2024-03-25T19:38:49.692 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:56.498 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.616+0000 7f25af006640 10 monclient: get_monmap_and_config 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.616+0000 7f25af006640 10 monclient: build_initial_monmap 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.616+0000 7f25af006640 10 monclient: monmap: 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:38:56.617305+0000 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:38:56.617305+0000 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:38:56.616 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.617+0000 7f25af006640 1 Processor -- start 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.617+0000 7f25af006640 1 -- start start 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.617+0000 7f25af006640 10 monclient: init 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.618+0000 7f25af006640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.618+0000 7f25af006640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.618+0000 7f25af006640 10 monclient(hunting): picked mon.noname-a con 0x7f25a80a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.618+0000 7f25af006640 10 monclient(hunting): picked mon.noname-c con 0x7f25a813f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.618+0000 7f25af006640 10 monclient(hunting): picked mon.noname-b con 0x7f25a81430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.618+0000 7f25af006640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a8142520 con 0x7f25a80a98f0 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.618+0000 7f25af006640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80b28a0 con 0x7f25a813f710 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.618+0000 7f25af006640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80b2a30 con 0x7f25a81430c0 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.618+0000 7f25af006640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.618+0000 7f25af006640 10 monclient(hunting): authenticate will time out at 4728.959015s 2024-03-25T19:38:56.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.627+0000 7f25a7fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f25a813f710 0x7f25a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:54986/0 (socket says 172.21.15.107:54986) 2024-03-25T19:38:56.627 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.627+0000 7f25a7fff640 1 -- 172.21.15.107:0/1076074599 learned_addr learned my addr 172.21.15.107:0/1076074599 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.640+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153599865 0 0) 0x7f25a80b2a30 con 0x7f25a81430c0 2024-03-25T19:38:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.640+0000 7f25a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.640+0000 7f25a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:56.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.640+0000 7f25a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:56.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.640+0000 7f25a77fe640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:38:56.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.640+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2598003520 con 0x7f25a81430c0 2024-03-25T19:38:56.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.642+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912485142 0 0) 0x7f25a8142520 con 0x7f25a80a98f0 2024-03-25T19:38:56.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.642+0000 7f25a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:56.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.642+0000 7f25a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:56.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.642+0000 7f25a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:56.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.642+0000 7f25a77fe640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:38:56.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.642+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a80b2a30 con 0x7f25a80a98f0 2024-03-25T19:38:56.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.644+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672703822 0 0) 0x7f25a80b28a0 con 0x7f25a813f710 2024-03-25T19:38:56.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.644+0000 7f25a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:56.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.644+0000 7f25a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:56.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.644+0000 7f25a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:56.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.644+0000 7f25a77fe640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:38:56.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.644+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a8142520 con 0x7f25a813f710 2024-03-25T19:38:56.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1258787629 0 0) 0x7f2598003520 con 0x7f25a81430c0 2024-03-25T19:38:56.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 >> v1:172.21.15.195:6789/0 conn(0x7f25a813f710 legacy=0x7f25a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:56.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 >> v1:172.21.15.107:6789/0 conn(0x7f25a80a98f0 legacy=0x7f25a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 1 monclient: found mon.noname-b 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a80b44c0 con 0x7f25a81430c0 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f25a0001a10 con 0x7f25a81430c0 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25af006640 5 monclient: authenticate success, global_id 4364 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25af006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.650+0000 7f25a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:56.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.651+0000 7f25a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.651+0000 7f25a77fe640 10 monclient: dump: 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:56.652 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.651+0000 7f25af006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:56.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25a0001b70 con 0x7f25a81430c0 2024-03-25T19:38:56.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:56.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25a77fe640 1 -- 172.21.15.107:0/1076074599 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f25a0001f80 con 0x7f25a81430c0 2024-03-25T19:38:56.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25af006640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:56.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25af006640 4 set_mon_vals no callback set 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25a77fe640 10 monclient: dump: 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:56.662 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:56.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25af006640 10 monclient: shutdown 2024-03-25T19:38:56.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25af006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:56.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25af006640 1 -- 172.21.15.107:0/1076074599 >> v1:172.21.15.107:6790/0 conn(0x7f25a81430c0 legacy=0x7f25a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:56.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25af006640 1 -- 172.21.15.107:0/1076074599 shutdown_connections 2024-03-25T19:38:56.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.662+0000 7f25af006640 1 -- 172.21.15.107:0/1076074599 >> 172.21.15.107:0/1076074599 conn(0x7f25a80b7ea0 msgr2=0x7f25a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:56.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.663+0000 7f25af006640 1 -- 172.21.15.107:0/1076074599 shutdown_connections 2024-03-25T19:38:56.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.663+0000 7f25af006640 1 -- 172.21.15.107:0/1076074599 wait complete. 2024-03-25T19:38:56.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.663+0000 7f25af006640 10 monclient: build_initial_monmap 2024-03-25T19:38:56.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.663+0000 7f25af006640 10 monclient: monmap: 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.664+0000 7f25af006640 1 librados: starting msgr at 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.664+0000 7f25af006640 1 librados: starting objecter 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.664+0000 7f25af006640 1 Processor -- start 2024-03-25T19:38:56.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.664+0000 7f25af006640 1 -- start start 2024-03-25T19:38:56.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.664+0000 7f25af006640 1 librados: setting wanted keys 2024-03-25T19:38:56.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.664+0000 7f25af006640 1 librados: calling monclient init 2024-03-25T19:38:56.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.664+0000 7f25af006640 10 monclient: init 2024-03-25T19:38:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.665+0000 7f25af006640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.665+0000 7f25af006640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.665+0000 7f25af006640 10 monclient(hunting): picked mon.noname-a con 0x7f25a80a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:38:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.665+0000 7f25af006640 10 monclient(hunting): picked mon.noname-b con 0x7f25a813f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:38:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.665+0000 7f25af006640 10 monclient(hunting): picked mon.noname-c con 0x7f25a81430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:38:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.665+0000 7f25af006640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a81396e0 con 0x7f25a80a98f0 2024-03-25T19:38:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.665+0000 7f25af006640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a8138530 con 0x7f25a813f710 2024-03-25T19:38:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.665+0000 7f25af006640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a81399c0 con 0x7f25a81430c0 2024-03-25T19:38:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.665+0000 7f25af006640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.665+0000 7f25af006640 10 monclient(hunting): authenticate will time out at 4729.006204s 2024-03-25T19:38:56.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.675+0000 7f25acda2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f25a80a98f0 0x7f25a8137820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52124/0 (socket says 172.21.15.107:52124) 2024-03-25T19:38:56.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.675+0000 7f25acda2640 1 -- 172.21.15.107:0/3327857163 learned_addr learned my addr 172.21.15.107:0/3327857163 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:38:56.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.676+0000 7f25a57fa640 10 client.?.objecter ms_handle_connect 0x7f25a80a98f0 2024-03-25T19:38:56.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.676+0000 7f25a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:56.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.679+0000 7f25a57fa640 10 client.?.objecter ms_handle_connect 0x7f25a813f710 2024-03-25T19:38:56.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.679+0000 7f25a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:56.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.685+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198007310 0 0) 0x7f25a81396e0 con 0x7f25a80a98f0 2024-03-25T19:38:56.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.685+0000 7f25a57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:56.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.685+0000 7f25a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:56.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.685+0000 7f25a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:56.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.685+0000 7f25a57fa640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:38:56.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.685+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f258c004eb0 con 0x7f25a80a98f0 2024-03-25T19:38:56.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.693+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284781847 0 0) 0x7f25a8138530 con 0x7f25a813f710 2024-03-25T19:38:56.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.693+0000 7f25a57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:56.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.693+0000 7f25a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:56.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.693+0000 7f25a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:56.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.693+0000 7f25a57fa640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:38:56.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.693+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a81396e0 con 0x7f25a813f710 2024-03-25T19:38:56.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.694+0000 7f25a57fa640 10 client.?.objecter ms_handle_connect 0x7f25a81430c0 2024-03-25T19:38:56.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.694+0000 7f25a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:56.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.699+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 180466558 0 0) 0x7f258c004eb0 con 0x7f25a80a98f0 2024-03-25T19:38:56.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.699+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25a8138530 con 0x7f25a80a98f0 2024-03-25T19:38:56.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.699+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f259c001a30 con 0x7f25a80a98f0 2024-03-25T19:38:56.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.699+0000 7f25a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:56.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.699+0000 7f25a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.699+0000 7f25a57fa640 10 monclient(hunting): dump: 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:56.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.699+0000 7f25a57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:56.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.704+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097735632 0 0) 0x7f25a81399c0 con 0x7f25a81430c0 2024-03-25T19:38:56.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.704+0000 7f25a57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:56.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.704+0000 7f25a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:56.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.704+0000 7f25a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:56.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.704+0000 7f25a57fa640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:38:56.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.704+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f258c004eb0 con 0x7f25a81430c0 2024-03-25T19:38:56.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.707+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2452890169 0 0) 0x7f25a81396e0 con 0x7f25a813f710 2024-03-25T19:38:56.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.707+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25a81399c0 con 0x7f25a813f710 2024-03-25T19:38:56.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.707+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f25940018b0 con 0x7f25a813f710 2024-03-25T19:38:56.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.707+0000 7f25a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:56.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.707+0000 7f25a57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:56.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.707+0000 7f25a57fa640 10 monclient(hunting): dump: 2024-03-25T19:38:56.707 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:56.707 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:56.707 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:56.708 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:56.708 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:56.708 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:56.708 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:56.708 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:56.708 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:56.708 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:56.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3094194445 0 0) 0x7f25a81399c0 con 0x7f25a813f710 2024-03-25T19:38:56.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 >> v1:172.21.15.107:6790/0 conn(0x7f25a81430c0 legacy=0x7f25a81e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:56.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 >> v1:172.21.15.107:6789/0 conn(0x7f25a80a98f0 legacy=0x7f25a8137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:56.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25a57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:38:56.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25a57fa640 1 monclient: found mon.b 2024-03-25T19:38:56.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:56.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25a57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:38:56.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a8138a50 con 0x7f25a813f710 2024-03-25T19:38:56.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25a57fa640 10 monclient: _finish_auth 0 2024-03-25T19:38:56.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25a57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:38:56.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:56.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25af006640 5 monclient: authenticate success, global_id 4354 2024-03-25T19:38:56.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25af006640 10 monclient: _renew_subs 2024-03-25T19:38:56.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25af006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:38:56.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25af006640 1 -- 172.21.15.107:0/3327857163 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a81380a0 con 0x7f25a813f710 2024-03-25T19:38:56.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25af006640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:56.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25af006640 10 monclient: _renew_subs 2024-03-25T19:38:56.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25af006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:38:56.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25af006640 1 -- 172.21.15.107:0/3327857163 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a81db140 con 0x7f25a813f710 2024-03-25T19:38:56.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.718+0000 7f25af006640 1 librados: init done 2024-03-25T19:38:56.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.719+0000 7f25af006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:56.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.719+0000 7f25af006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:56.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.719+0000 7f25af006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:38:56.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:56.719+0000 7f25af006640 1 -- 172.21.15.107:0/3327857163 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a81383a0 con 0x7f25a813f710 2024-03-25T19:38:57.000 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.001+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2594001d70 con 0x7f25a813f710 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.001+0000 7f25a57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.001+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2594004a00 con 0x7f25a813f710 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.001+0000 7f25a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.001+0000 7f25a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.001+0000 7f25a57fa640 10 monclient: dump: 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:38:57.001 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:38:57.002 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:38:57.002 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:38:57.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.001+0000 7f25a6ffd640 4 set_mon_vals no callback set 2024-03-25T19:38:57.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.001+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f2594004c20 con 0x7f25a813f710 2024-03-25T19:38:57.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.002+0000 7f25acda2640 10 monclient: get_auth_request con 0x7f258c06df70 auth_method 0 2024-03-25T19:38:57.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.002+0000 7f25a57fa640 10 client.4354.objecter ms_handle_connect 0x7f258c06df70 2024-03-25T19:38:57.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.009+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f25940547d0 con 0x7f25a813f710 2024-03-25T19:38:57.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.009+0000 7f25a57fa640 10 client.4354.objecter ms_dispatch 0x7f25a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:57.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.009+0000 7f25a57fa640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:57.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.009+0000 7f25a57fa640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:57.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.009+0000 7f25a57fa640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T19:38:57.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.010+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2594050110 con 0x7f25a813f710 2024-03-25T19:38:57.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.010+0000 7f25a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:57.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.010+0000 7f25a57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:57.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.256+0000 7f25af006640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:57.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.256+0000 7f25af006640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:57.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.256+0000 7f25af006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:38:57.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.256+0000 7f25af006640 1 -- 172.21.15.107:0/3327857163 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25a80627b0 con 0x7f25a813f710 2024-03-25T19:38:57.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.339+0000 7f25a57fa640 1 -- 172.21.15.107:0/3327857163 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f2594052210 con 0x7f25a813f710 2024-03-25T19:38:57.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.339+0000 7f25a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:57.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.339+0000 7f25a57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:57.339 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:38:57.339 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.341+0000 7f25af006640 10 librados: watch_flush enter 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 10 librados: watch_flush exit 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 10 client.4354.objecter successfully canceled tick 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 1 -- 172.21.15.107:0/3327857163 >> v1:172.21.15.195:6800/2984408421 conn(0x7f258c06df70 legacy=0x7f258c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 10 monclient: shutdown 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 1 -- 172.21.15.107:0/3327857163 >> v1:172.21.15.195:6789/0 conn(0x7f25a813f710 legacy=0x7f25a81da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 1 -- 172.21.15.107:0/3327857163 shutdown_connections 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 1 -- 172.21.15.107:0/3327857163 >> 172.21.15.107:0/3327857163 conn(0x7f25a80b7ea0 msgr2=0x7f25a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 1 -- 172.21.15.107:0/3327857163 shutdown_connections 2024-03-25T19:38:57.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 1 -- 172.21.15.107:0/3327857163 wait complete. 2024-03-25T19:38:57.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:38:57.342+0000 7f25af006640 1 librados: shutdown 2024-03-25T19:38:57.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:04.161 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:04.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.265+0000 7ff2003f4640 10 monclient: get_monmap_and_config 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.265+0000 7ff2003f4640 10 monclient: build_initial_monmap 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.265+0000 7ff2003f4640 10 monclient: monmap: 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:39:04.266177+0000 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:39:04.266177+0000 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:39:04.265 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:04.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.265+0000 7ff2003f4640 1 Processor -- start 2024-03-25T19:39:04.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.265+0000 7ff2003f4640 1 -- start start 2024-03-25T19:39:04.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.265+0000 7ff2003f4640 10 monclient: init 2024-03-25T19:39:04.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.266+0000 7ff2003f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:04.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.266+0000 7ff2003f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:04.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.266+0000 7ff2003f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff1f80b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:04.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.266+0000 7ff2003f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff1f80b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:04.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.266+0000 7ff2003f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff1f814be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:04.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.266+0000 7ff2003f4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f80af940 con 0x7ff1f80b1730 2024-03-25T19:39:04.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.266+0000 7ff2003f4640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f80b0a70 con 0x7ff1f814be90 2024-03-25T19:39:04.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.266+0000 7ff2003f4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f80b0d50 con 0x7ff1f80b4d60 2024-03-25T19:39:04.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.266+0000 7ff2003f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:04.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.266+0000 7ff2003f4640 10 monclient(hunting): authenticate will time out at 4736.607652s 2024-03-25T19:39:04.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.272+0000 7ff1fe991640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff1f814be90 0x7ff1f814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:54410/0 (socket says 172.21.15.107:54410) 2024-03-25T19:39:04.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.272+0000 7ff1fe991640 1 -- 172.21.15.107:0/3660282339 learned_addr learned my addr 172.21.15.107:0/3660282339 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:04.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.279+0000 7ff1fd18e640 1 -- 172.21.15.107:0/3660282339 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891710547 0 0) 0x7ff1f80b0a70 con 0x7ff1f814be90 2024-03-25T19:39:04.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.279+0000 7ff1fd18e640 10 monclient(hunting): _negotiate 2024-03-25T19:39:04.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.279+0000 7ff1fd18e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:04.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.279+0000 7ff1fd18e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:04.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.279+0000 7ff1fd18e640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:39:04.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.279+0000 7ff1fd18e640 1 -- 172.21.15.107:0/3660282339 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e4003520 con 0x7ff1f814be90 2024-03-25T19:39:04.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 1 -- 172.21.15.107:0/3660282339 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2686933941 0 0) 0x7ff1e4003520 con 0x7ff1f814be90 2024-03-25T19:39:04.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 1 -- 172.21.15.107:0/3660282339 >> v1:172.21.15.107:6790/0 conn(0x7ff1f80b4d60 legacy=0x7ff1f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:39:04.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 1 -- 172.21.15.107:0/3660282339 >> v1:172.21.15.107:6789/0 conn(0x7ff1f80b1730 legacy=0x7ff1f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:04.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 10 monclient: _finish_hunting 0 2024-03-25T19:39:04.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 1 monclient: found mon.noname-c 2024-03-25T19:39:04.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:04.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:39:04.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 1 -- 172.21.15.107:0/3660282339 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f80b1030 con 0x7ff1f814be90 2024-03-25T19:39:04.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 10 monclient: _finish_auth 0 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 10 monclient: _check_auth_tickets 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 1 -- 172.21.15.107:0/3660282339 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff1f0001a10 con 0x7ff1f814be90 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff1fd18e640 10 monclient: dump: 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:04.287 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:04.288 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:04.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff2003f4640 5 monclient: authenticate success, global_id 4357 2024-03-25T19:39:04.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.286+0000 7ff2003f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:04.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.291+0000 7ff1fd18e640 1 -- 172.21.15.107:0/3660282339 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f0001be0 con 0x7ff1f814be90 2024-03-25T19:39:04.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.291+0000 7ff1fd18e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:04.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.291+0000 7ff2003f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:04.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.291+0000 7ff2003f4640 4 set_mon_vals no callback set 2024-03-25T19:39:04.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.292+0000 7ff2003f4640 10 monclient: shutdown 2024-03-25T19:39:04.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.292+0000 7ff2003f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:04.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.292+0000 7ff2003f4640 1 -- 172.21.15.107:0/3660282339 >> v1:172.21.15.195:6789/0 conn(0x7ff1f814be90 legacy=0x7ff1f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.292+0000 7ff2003f4640 1 -- 172.21.15.107:0/3660282339 shutdown_connections 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.292+0000 7ff2003f4640 1 -- 172.21.15.107:0/3660282339 >> 172.21.15.107:0/3660282339 conn(0x7ff1f80617a0 msgr2=0x7ff1f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.292+0000 7ff2003f4640 1 -- 172.21.15.107:0/3660282339 shutdown_connections 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.292+0000 7ff2003f4640 1 -- 172.21.15.107:0/3660282339 wait complete. 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.293+0000 7ff2003f4640 10 monclient: build_initial_monmap 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.293+0000 7ff2003f4640 10 monclient: monmap: 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:39:04.297 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.293+0000 7ff2003f4640 1 librados: starting msgr at 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.293+0000 7ff2003f4640 1 librados: starting objecter 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.293+0000 7ff2003f4640 1 Processor -- start 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 1 -- start start 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 1 librados: setting wanted keys 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 1 librados: calling monclient init 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 10 monclient: init 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:04.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:04.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff1f80b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:04.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff1f80b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:04.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff1f814be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:04.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f8141200 con 0x7ff1f814be90 2024-03-25T19:39:04.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f8143130 con 0x7ff1f80b4d60 2024-03-25T19:39:04.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f81432c0 con 0x7ff1f80b1730 2024-03-25T19:39:04.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:04.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.294+0000 7ff2003f4640 10 monclient(hunting): authenticate will time out at 4736.635731s 2024-03-25T19:39:04.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.309+0000 7ff1fd98f640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff1f80b4d60 0x7ff1f8132d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:54446/0 (socket says 172.21.15.107:54446) 2024-03-25T19:39:04.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.309+0000 7ff1fd98f640 1 -- 172.21.15.107:0/1135657246 learned_addr learned my addr 172.21.15.107:0/1135657246 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:04.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.312+0000 7ff1eeffd640 10 client.?.objecter ms_handle_connect 0x7ff1f80b4d60 2024-03-25T19:39:04.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.312+0000 7ff1eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:04.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.317+0000 7ff1eeffd640 10 client.?.objecter ms_handle_connect 0x7ff1f814be90 2024-03-25T19:39:04.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.317+0000 7ff1eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:04.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.317+0000 7ff1eeffd640 10 client.?.objecter ms_handle_connect 0x7ff1f80b1730 2024-03-25T19:39:04.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.317+0000 7ff1eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:04.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.325+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116809182 0 0) 0x7ff1f81432c0 con 0x7ff1f80b1730 2024-03-25T19:39:04.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.325+0000 7ff1eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:04.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.325+0000 7ff1eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:04.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.325+0000 7ff1eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:04.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.325+0000 7ff1eeffd640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:39:04.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.325+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1dc004eb0 con 0x7ff1f80b1730 2024-03-25T19:39:04.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.327+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684310896 0 0) 0x7ff1f8143130 con 0x7ff1f80b4d60 2024-03-25T19:39:04.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.327+0000 7ff1eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:04.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.327+0000 7ff1eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:04.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.327+0000 7ff1eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:04.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.327+0000 7ff1eeffd640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:39:04.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.327+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f81432c0 con 0x7ff1f80b4d60 2024-03-25T19:39:04.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.343+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418938058 0 0) 0x7ff1f8141200 con 0x7ff1f814be90 2024-03-25T19:39:04.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.343+0000 7ff1eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:04.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.343+0000 7ff1eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:04.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.343+0000 7ff1eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:04.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.343+0000 7ff1eeffd640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:39:04.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.343+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f8143130 con 0x7ff1f814be90 2024-03-25T19:39:04.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.350+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 313281378 0 0) 0x7ff1f81432c0 con 0x7ff1f80b4d60 2024-03-25T19:39:04.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.350+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1f8141200 con 0x7ff1f80b4d60 2024-03-25T19:39:04.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.350+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff1e80019b0 con 0x7ff1f80b4d60 2024-03-25T19:39:04.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.350+0000 7ff1eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:04.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.350+0000 7ff1eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:04.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.350+0000 7ff1eeffd640 10 monclient(hunting): dump: 2024-03-25T19:39:04.350 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:04.350 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:04.350 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:04.351 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:04.351 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:04.351 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:04.351 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:04.351 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:04.351 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:04.351 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:04.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.350+0000 7ff1eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:04.356 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.356+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 270155323 0 0) 0x7ff1dc004eb0 con 0x7ff1f80b1730 2024-03-25T19:39:04.356 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.356+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1f81432c0 con 0x7ff1f80b1730 2024-03-25T19:39:04.356 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.356+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff1f4001510 con 0x7ff1f80b1730 2024-03-25T19:39:04.356 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.356+0000 7ff1eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:04.356 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.356+0000 7ff1eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:04.356 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.356+0000 7ff1eeffd640 10 monclient(hunting): dump: 2024-03-25T19:39:04.356 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:04.357 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:04.357 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:04.357 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:04.357 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:04.357 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:04.357 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:04.357 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:04.357 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:04.357 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:04.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 821819597 0 0) 0x7ff1f8143130 con 0x7ff1f814be90 2024-03-25T19:39:04.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1dc004eb0 con 0x7ff1f814be90 2024-03-25T19:39:04.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff1f0001ef0 con 0x7ff1f814be90 2024-03-25T19:39:04.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:04.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:04.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 10 monclient(hunting): dump: 2024-03-25T19:39:04.370 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:04.370 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:04.370 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:04.370 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:04.370 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:04.370 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:04.370 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3887443383 0 0) 0x7ff1f8141200 con 0x7ff1f80b4d60 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 >> v1:172.21.15.107:6790/0 conn(0x7ff1f80b1730 legacy=0x7ff1f81d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 >> v1:172.21.15.107:6789/0 conn(0x7ff1f814be90 legacy=0x7ff1f81e2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 1 monclient: found mon.b 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f8140d40 con 0x7ff1f80b4d60 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:04.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff1eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff2003f4640 5 monclient: authenticate success, global_id 4360 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff2003f4640 10 monclient: _renew_subs 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff2003f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff2003f4640 1 -- 172.21.15.107:0/1135657246 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f8140380 con 0x7ff1f80b4d60 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff2003f4640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff2003f4640 10 monclient: _renew_subs 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff2003f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff2003f4640 1 -- 172.21.15.107:0/1135657246 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f81334a0 con 0x7ff1f80b4d60 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.368+0000 7ff2003f4640 1 librados: init done 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.370+0000 7ff2003f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:04.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.370+0000 7ff2003f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:04.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.370+0000 7ff2003f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:04.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.370+0000 7ff2003f4640 1 -- 172.21.15.107:0/1135657246 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f81337f0 con 0x7ff1f80b4d60 2024-03-25T19:39:04.662 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.662+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1e8003610 con 0x7ff1f80b4d60 2024-03-25T19:39:04.662 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.662+0000 7ff1eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:04.662 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.662+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff1e8003920 con 0x7ff1f80b4d60 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.662+0000 7ff1eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.662+0000 7ff1eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.662+0000 7ff1eeffd640 10 monclient: dump: 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:04.663 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:04.664 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:04.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.662+0000 7ff1fc98d640 4 set_mon_vals no callback set 2024-03-25T19:39:04.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.663+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7ff1e8003b40 con 0x7ff1f80b4d60 2024-03-25T19:39:04.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.664+0000 7ff1fe190640 10 monclient: get_auth_request con 0x7ff1dc06e0b0 auth_method 0 2024-03-25T19:39:04.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.664+0000 7ff1eeffd640 10 client.4360.objecter ms_handle_connect 0x7ff1dc06e0b0 2024-03-25T19:39:04.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.668+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7ff1e8054800 con 0x7ff1f80b4d60 2024-03-25T19:39:04.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.668+0000 7ff1eeffd640 10 client.4360.objecter ms_dispatch 0x7ff1f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:04.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.668+0000 7ff1eeffd640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:04.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.668+0000 7ff1eeffd640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:04.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.668+0000 7ff1eeffd640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T19:39:04.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.668+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1e8050310 con 0x7ff1f80b4d60 2024-03-25T19:39:04.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.668+0000 7ff1eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:04.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.668+0000 7ff1eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:04.790 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:03:35.106 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:04.797 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:03:35.114 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:04.834 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:03:35.121 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:04.836 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:03:35.152 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:04.836 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:03:35.152 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:04.840 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:03:35.127 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:04.847 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:03:35.163 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:04.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.886+0000 7ff2003f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:04.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.886+0000 7ff2003f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:04.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.886+0000 7ff2003f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:04.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.886+0000 7ff2003f4640 1 -- 172.21.15.107:0/1135657246 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1f80627b0 con 0x7ff1f80b4d60 2024-03-25T19:39:04.893 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:03:35.180 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:04.895 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:03:35.212 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:04.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.982+0000 7ff1eeffd640 1 -- 172.21.15.107:0/1135657246 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7ff1e8052240 con 0x7ff1f80b4d60 2024-03-25T19:39:04.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.982+0000 7ff1eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:04.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.982+0000 7ff1eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:04.982 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:39:04.982 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:04.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 10 librados: watch_flush enter 2024-03-25T19:39:04.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 10 librados: watch_flush exit 2024-03-25T19:39:04.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T19:39:04.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 10 client.4360.objecter successfully canceled tick 2024-03-25T19:39:04.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 1 -- 172.21.15.107:0/1135657246 >> v1:172.21.15.195:6800/2984408421 conn(0x7ff1dc06e0b0 legacy=0x7ff1dc070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:04.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 10 monclient: shutdown 2024-03-25T19:39:04.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:04.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 1 -- 172.21.15.107:0/1135657246 >> v1:172.21.15.195:6789/0 conn(0x7ff1f80b4d60 legacy=0x7ff1f8132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:04.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 1 -- 172.21.15.107:0/1135657246 shutdown_connections 2024-03-25T19:39:04.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 1 -- 172.21.15.107:0/1135657246 >> 172.21.15.107:0/1135657246 conn(0x7ff1f80617a0 msgr2=0x7ff1f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:04.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 1 -- 172.21.15.107:0/1135657246 shutdown_connections 2024-03-25T19:39:04.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 1 -- 172.21.15.107:0/1135657246 wait complete. 2024-03-25T19:39:04.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:04.984+0000 7ff2003f4640 1 librados: shutdown 2024-03-25T19:39:04.994 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:11.800 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:11.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.914+0000 7fc9ee554640 10 monclient: get_monmap_and_config 2024-03-25T19:39:11.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.914+0000 7fc9ee554640 10 monclient: build_initial_monmap 2024-03-25T19:39:11.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.915+0000 7fc9ee554640 10 monclient: monmap: 2024-03-25T19:39:11.914 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:11.914 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:11.915 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:39:11.915961+0000 2024-03-25T19:39:11.915 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:39:11.915961+0000 2024-03-25T19:39:11.915 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:11.915 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:11.915 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:11.915 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:39:11.915 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:39:11.915 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:11.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.915+0000 7fc9ee554640 1 Processor -- start 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.915+0000 7fc9ee554640 1 -- start start 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.915+0000 7fc9ee554640 10 monclient: init 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.916+0000 7fc9ee554640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.916+0000 7fc9ee554640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.916+0000 7fc9ee554640 10 monclient(hunting): picked mon.noname-a con 0x7fc9e80a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.916+0000 7fc9ee554640 10 monclient(hunting): picked mon.noname-b con 0x7fc9e813f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.916+0000 7fc9ee554640 10 monclient(hunting): picked mon.noname-c con 0x7fc9e81430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.916+0000 7fc9ee554640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e8142520 con 0x7fc9e80a98f0 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.916+0000 7fc9ee554640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e80b28a0 con 0x7fc9e81430c0 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.917+0000 7fc9ee554640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e80b2a30 con 0x7fc9e813f710 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.917+0000 7fc9ee554640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:11.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.917+0000 7fc9ee554640 10 monclient(hunting): authenticate will time out at 4744.257865s 2024-03-25T19:39:11.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.933+0000 7fc9e77fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc9e813f710 0x7fc9e81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49520/0 (socket says 172.21.15.107:49520) 2024-03-25T19:39:11.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.933+0000 7fc9e77fe640 1 -- 172.21.15.107:0/1318738385 learned_addr learned my addr 172.21.15.107:0/1318738385 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:11.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.952+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445903266 0 0) 0x7fc9e80b28a0 con 0x7fc9e81430c0 2024-03-25T19:39:11.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.952+0000 7fc9e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:11.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.952+0000 7fc9e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:11.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.952+0000 7fc9e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:11.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.952+0000 7fc9e6ffd640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:39:11.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.952+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d0003520 con 0x7fc9e81430c0 2024-03-25T19:39:11.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.960+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011823917 0 0) 0x7fc9e80b2a30 con 0x7fc9e813f710 2024-03-25T19:39:11.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.960+0000 7fc9e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:11.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.960+0000 7fc9e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:11.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.960+0000 7fc9e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:11.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.960+0000 7fc9e6ffd640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:39:11.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.960+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e80b28a0 con 0x7fc9e813f710 2024-03-25T19:39:11.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.965+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893456483 0 0) 0x7fc9e8142520 con 0x7fc9e80a98f0 2024-03-25T19:39:11.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.965+0000 7fc9e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:11.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.965+0000 7fc9e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:11.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.965+0000 7fc9e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:11.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.965+0000 7fc9e6ffd640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:39:11.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.965+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e80b2a30 con 0x7fc9e80a98f0 2024-03-25T19:39:11.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.972+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3263316379 0 0) 0x7fc9e80b28a0 con 0x7fc9e813f710 2024-03-25T19:39:11.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.972+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 >> v1:172.21.15.195:6789/0 conn(0x7fc9e81430c0 legacy=0x7fc9e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 >> v1:172.21.15.107:6789/0 conn(0x7fc9e80a98f0 legacy=0x7fc9e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 1 monclient: found mon.noname-b 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e80b44c0 con 0x7fc9e813f710 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fc9dc001670 con 0x7fc9e813f710 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9e6ffd640 10 monclient: dump: 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:11.973 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:11.974 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:11.974 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:11.974 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:11.974 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:11.974 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:11.974 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:11.974 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:11.974 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:11.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9ee554640 5 monclient: authenticate success, global_id 4373 2024-03-25T19:39:11.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.973+0000 7fc9ee554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:11.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.990+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9dc0017d0 con 0x7fc9e813f710 2024-03-25T19:39:11.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.990+0000 7fc9e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:11.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.990+0000 7fc9e6ffd640 1 -- 172.21.15.107:0/1318738385 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fc9dc001be0 con 0x7fc9e813f710 2024-03-25T19:39:11.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.990+0000 7fc9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:11.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.990+0000 7fc9e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:11.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.990+0000 7fc9e6ffd640 10 monclient: dump: 2024-03-25T19:39:11.990 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:11.990 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:11.990 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:11.990 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:11.990 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:11.991 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:11.991 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:11.991 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:11.991 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:11.991 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:11.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.990+0000 7fc9ee554640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:11.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.990+0000 7fc9ee554640 4 set_mon_vals no callback set 2024-03-25T19:39:11.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.991+0000 7fc9ee554640 10 monclient: shutdown 2024-03-25T19:39:11.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.991+0000 7fc9ee554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:11.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.991+0000 7fc9ee554640 1 -- 172.21.15.107:0/1318738385 >> v1:172.21.15.107:6790/0 conn(0x7fc9e813f710 legacy=0x7fc9e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:11.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.991+0000 7fc9ee554640 1 -- 172.21.15.107:0/1318738385 shutdown_connections 2024-03-25T19:39:11.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.991+0000 7fc9ee554640 1 -- 172.21.15.107:0/1318738385 >> 172.21.15.107:0/1318738385 conn(0x7fc9e80b7ea0 msgr2=0x7fc9e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:11.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.991+0000 7fc9ee554640 1 -- 172.21.15.107:0/1318738385 shutdown_connections 2024-03-25T19:39:11.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.991+0000 7fc9ee554640 1 -- 172.21.15.107:0/1318738385 wait complete. 2024-03-25T19:39:11.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.992+0000 7fc9ee554640 10 monclient: build_initial_monmap 2024-03-25T19:39:11.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.992+0000 7fc9ee554640 10 monclient: monmap: 2024-03-25T19:39:11.993 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.992+0000 7fc9ee554640 1 librados: starting msgr at 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.992+0000 7fc9ee554640 1 librados: starting objecter 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.992+0000 7fc9ee554640 1 Processor -- start 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.992+0000 7fc9ee554640 1 -- start start 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 1 librados: setting wanted keys 2024-03-25T19:39:11.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 1 librados: calling monclient init 2024-03-25T19:39:11.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 10 monclient: init 2024-03-25T19:39:11.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:11.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:11.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 10 monclient(hunting): picked mon.noname-c con 0x7fc9e80a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:11.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 10 monclient(hunting): picked mon.noname-b con 0x7fc9e813f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:11.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 10 monclient(hunting): picked mon.noname-a con 0x7fc9e81430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:11.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e8136fa0 con 0x7fc9e81430c0 2024-03-25T19:39:11.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e813b780 con 0x7fc9e813f710 2024-03-25T19:39:11.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e81d7ba0 con 0x7fc9e80a98f0 2024-03-25T19:39:11.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:11.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.993+0000 7fc9ee554640 10 monclient(hunting): authenticate will time out at 4744.334607s 2024-03-25T19:39:11.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.998+0000 7fc9e77fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc9e813f710 0x7fc9e81d7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:56224/0 (socket says 172.21.15.107:56224) 2024-03-25T19:39:11.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.998+0000 7fc9e77fe640 1 -- 172.21.15.107:0/3721223126 learned_addr learned my addr 172.21.15.107:0/3721223126 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:11.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.999+0000 7fc9e4ff9640 10 client.?.objecter ms_handle_connect 0x7fc9e813f710 2024-03-25T19:39:11.999 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:11.999+0000 7fc9e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:12.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.007+0000 7fc9e4ff9640 10 client.?.objecter ms_handle_connect 0x7fc9e81430c0 2024-03-25T19:39:12.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.007+0000 7fc9e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:12.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.012+0000 7fc9e4ff9640 10 client.?.objecter ms_handle_connect 0x7fc9e80a98f0 2024-03-25T19:39:12.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.012+0000 7fc9e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:12.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.014+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641543622 0 0) 0x7fc9e813b780 con 0x7fc9e813f710 2024-03-25T19:39:12.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.014+0000 7fc9e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:12.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.014+0000 7fc9e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:12.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.014+0000 7fc9e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:12.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.014+0000 7fc9e4ff9640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:39:12.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.014+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d4004eb0 con 0x7fc9e813f710 2024-03-25T19:39:12.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.030+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847506562 0 0) 0x7fc9e81d7ba0 con 0x7fc9e80a98f0 2024-03-25T19:39:12.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.030+0000 7fc9e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:12.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.030+0000 7fc9e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:12.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.030+0000 7fc9e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:12.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.030+0000 7fc9e4ff9640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:39:12.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.030+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e813b780 con 0x7fc9e80a98f0 2024-03-25T19:39:12.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.110+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4091278029 0 0) 0x7fc9e813b780 con 0x7fc9e80a98f0 2024-03-25T19:39:12.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.110+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9e81d7ba0 con 0x7fc9e80a98f0 2024-03-25T19:39:12.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.110+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fc9d8001870 con 0x7fc9e80a98f0 2024-03-25T19:39:12.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.110+0000 7fc9e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.110+0000 7fc9e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.110+0000 7fc9e4ff9640 10 monclient(hunting): dump: 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:12.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.110+0000 7fc9e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:12.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1913693818 0 0) 0x7fc9e81d7ba0 con 0x7fc9e80a98f0 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 >> v1:172.21.15.195:6789/0 conn(0x7fc9e813f710 legacy=0x7fc9e81d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 >> v1:172.21.15.107:6789/0 conn(0x7fc9e81430c0 legacy=0x7fc9e81e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9e4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9e4ff9640 1 monclient: found mon.c 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9e4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e8137990 con 0x7fc9e80a98f0 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9e4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9e4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9ee554640 5 monclient: authenticate success, global_id 4376 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9ee554640 10 monclient: _renew_subs 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9ee554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:39:12.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9ee554640 1 -- 172.21.15.107:0/3721223126 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e8137c40 con 0x7fc9e80a98f0 2024-03-25T19:39:12.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9ee554640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:12.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9ee554640 10 monclient: _renew_subs 2024-03-25T19:39:12.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9ee554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:39:12.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9ee554640 1 -- 172.21.15.107:0/3721223126 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e8139860 con 0x7fc9e80a98f0 2024-03-25T19:39:12.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.124+0000 7fc9ee554640 1 librados: init done 2024-03-25T19:39:12.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.125+0000 7fc9ee554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:12.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.125+0000 7fc9ee554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:12.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.125+0000 7fc9ee554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:39:12.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.125+0000 7fc9ee554640 1 -- 172.21.15.107:0/3721223126 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e81375f0 con 0x7fc9e80a98f0 2024-03-25T19:39:12.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.420+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9d8003610 con 0x7fc9e80a98f0 2024-03-25T19:39:12.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.420+0000 7fc9e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:12.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.420+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fc9d8001d80 con 0x7fc9e80a98f0 2024-03-25T19:39:12.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.420+0000 7fc9e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:12.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.420+0000 7fc9e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:12.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.420+0000 7fc9e4ff9640 10 monclient: dump: 2024-03-25T19:39:12.420 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:12.420 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:12.420 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:12.420 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:12.421 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:12.421 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:12.421 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:12.421 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:12.421 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:12.421 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:12.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.420+0000 7fc9e67fc640 4 set_mon_vals no callback set 2024-03-25T19:39:12.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.421+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fc9d8003b20 con 0x7fc9e80a98f0 2024-03-25T19:39:12.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.422+0000 7fc9e77fe640 10 monclient: get_auth_request con 0x7fc9d406dd20 auth_method 0 2024-03-25T19:39:12.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.422+0000 7fc9e4ff9640 10 client.4376.objecter ms_handle_connect 0x7fc9d406dd20 2024-03-25T19:39:12.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.427+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fc9d8053590 con 0x7fc9e80a98f0 2024-03-25T19:39:12.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.427+0000 7fc9e4ff9640 10 client.4376.objecter ms_dispatch 0x7fc9e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:12.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.427+0000 7fc9e4ff9640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:12.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.427+0000 7fc9e4ff9640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:12.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.427+0000 7fc9e4ff9640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-25T19:39:12.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.428+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9d804f0a0 con 0x7fc9e80a98f0 2024-03-25T19:39:12.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.428+0000 7fc9e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:12.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.428+0000 7fc9e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:12.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.686+0000 7fc9ee554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:12.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.686+0000 7fc9ee554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:12.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.686+0000 7fc9ee554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:39:12.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.686+0000 7fc9ee554640 1 -- 172.21.15.107:0/3721223126 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9e80627b0 con 0x7fc9e80a98f0 2024-03-25T19:39:12.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.736+0000 7fc9e4ff9640 1 -- 172.21.15.107:0/3721223126 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fc9d8050fd0 con 0x7fc9e80a98f0 2024-03-25T19:39:12.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.736+0000 7fc9e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:12.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.736+0000 7fc9e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:12.737 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:39:12.737 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:12.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 10 librados: watch_flush enter 2024-03-25T19:39:12.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 10 librados: watch_flush exit 2024-03-25T19:39:12.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-25T19:39:12.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 10 client.4376.objecter successfully canceled tick 2024-03-25T19:39:12.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 1 -- 172.21.15.107:0/3721223126 >> v1:172.21.15.195:6800/2984408421 conn(0x7fc9d406dd20 legacy=0x7fc9d40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:12.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 10 monclient: shutdown 2024-03-25T19:39:12.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:12.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 1 -- 172.21.15.107:0/3721223126 >> v1:172.21.15.107:6790/0 conn(0x7fc9e80a98f0 legacy=0x7fc9e8136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:12.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 1 -- 172.21.15.107:0/3721223126 shutdown_connections 2024-03-25T19:39:12.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 1 -- 172.21.15.107:0/3721223126 >> 172.21.15.107:0/3721223126 conn(0x7fc9e80b7ea0 msgr2=0x7fc9e814ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:12.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 1 -- 172.21.15.107:0/3721223126 shutdown_connections 2024-03-25T19:39:12.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 1 -- 172.21.15.107:0/3721223126 wait complete. 2024-03-25T19:39:12.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:12.739+0000 7fc9ee554640 1 librados: shutdown 2024-03-25T19:39:12.751 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:19.558 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:19.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.674+0000 7fbd1e756640 10 monclient: get_monmap_and_config 2024-03-25T19:39:19.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.674+0000 7fbd1e756640 10 monclient: build_initial_monmap 2024-03-25T19:39:19.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.674+0000 7fbd1e756640 10 monclient: monmap: 2024-03-25T19:39:19.674 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:19.674 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:19.674 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:39:19.675134+0000 2024-03-25T19:39:19.674 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:39:19.675134+0000 2024-03-25T19:39:19.674 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:19.674 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:19.674 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:19.674 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:39:19.674 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:39:19.674 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:19.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 1 Processor -- start 2024-03-25T19:39:19.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 1 -- start start 2024-03-25T19:39:19.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 10 monclient: init 2024-03-25T19:39:19.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:19.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:19.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 10 monclient(hunting): picked mon.noname-c con 0x7fbd180b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:19.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 10 monclient(hunting): picked mon.noname-a con 0x7fbd180b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:19.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 10 monclient(hunting): picked mon.noname-b con 0x7fbd1814be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:19.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180af940 con 0x7fbd180b1730 2024-03-25T19:39:19.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180b0a70 con 0x7fbd180b4d60 2024-03-25T19:39:19.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180b0d50 con 0x7fbd1814be90 2024-03-25T19:39:19.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:19.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.675+0000 7fbd1e756640 10 monclient(hunting): authenticate will time out at 4752.016683s 2024-03-25T19:39:19.684 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.685+0000 7fbd1ccf3640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fbd1814be90 0x7fbd1814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54570/0 (socket says 172.21.15.107:54570) 2024-03-25T19:39:19.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.685+0000 7fbd1ccf3640 1 -- 172.21.15.107:0/2418736608 learned_addr learned my addr 172.21.15.107:0/2418736608 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:19.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176811445 0 0) 0x7fbd180b0a70 con 0x7fbd180b4d60 2024-03-25T19:39:19.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd00003520 con 0x7fbd180b4d60 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629198287 0 0) 0x7fbd180b0d50 con 0x7fbd1814be90 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd180b0a70 con 0x7fbd1814be90 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945888009 0 0) 0x7fbd180af940 con 0x7fbd180b1730 2024-03-25T19:39:19.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:19.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:19.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:19.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:39:19.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.709+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd180b0d50 con 0x7fbd180b1730 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3047985519 0 0) 0x7fbd180b0a70 con 0x7fbd1814be90 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 >> v1:172.21.15.195:6789/0 conn(0x7fbd180b4d60 legacy=0x7fbd1813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 >> v1:172.21.15.107:6789/0 conn(0x7fbd180b1730 legacy=0x7fbd180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 1 monclient: found mon.noname-b 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd180b1030 con 0x7fbd1814be90 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fbd100019f0 con 0x7fbd1814be90 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd1e756640 5 monclient: authenticate success, global_id 4379 2024-03-25T19:39:19.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd1e756640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd16ffd640 10 monclient: dump: 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:19.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.732+0000 7fbd1e756640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:19.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd10001b50 con 0x7fbd1814be90 2024-03-25T19:39:19.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:19.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd16ffd640 1 -- 172.21.15.107:0/2418736608 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fbd10001f60 con 0x7fbd1814be90 2024-03-25T19:39:19.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd16ffd640 10 monclient: dump: 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd1e756640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:19.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd1e756640 4 set_mon_vals no callback set 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd1e756640 10 monclient: shutdown 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd1e756640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd1e756640 1 -- 172.21.15.107:0/2418736608 >> v1:172.21.15.107:6790/0 conn(0x7fbd1814be90 legacy=0x7fbd1814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd1e756640 1 -- 172.21.15.107:0/2418736608 shutdown_connections 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.764+0000 7fbd1e756640 1 -- 172.21.15.107:0/2418736608 >> 172.21.15.107:0/2418736608 conn(0x7fbd180617a0 msgr2=0x7fbd18061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.765+0000 7fbd1e756640 1 -- 172.21.15.107:0/2418736608 shutdown_connections 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.765+0000 7fbd1e756640 1 -- 172.21.15.107:0/2418736608 wait complete. 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.765+0000 7fbd1e756640 10 monclient: build_initial_monmap 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.765+0000 7fbd1e756640 10 monclient: monmap: 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:39:19.767 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.765+0000 7fbd1e756640 1 librados: starting msgr at 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.765+0000 7fbd1e756640 1 librados: starting objecter 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.765+0000 7fbd1e756640 1 Processor -- start 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 1 -- start start 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 1 librados: setting wanted keys 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 1 librados: calling monclient init 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 10 monclient: init 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:19.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:19.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 10 monclient(hunting): picked mon.noname-b con 0x7fbd180b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:19.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 10 monclient(hunting): picked mon.noname-c con 0x7fbd180b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:19.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 10 monclient(hunting): picked mon.noname-a con 0x7fbd1814be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:19.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd181384b0 con 0x7fbd1814be90 2024-03-25T19:39:19.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd181327e0 con 0x7fbd180b1730 2024-03-25T19:39:19.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd18137e70 con 0x7fbd180b4d60 2024-03-25T19:39:19.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:19.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.766+0000 7fbd1e756640 10 monclient(hunting): authenticate will time out at 4752.107612s 2024-03-25T19:39:19.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.780+0000 7fbd1ccf3640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbd1814be90 0x7fbd181320d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44558/0 (socket says 172.21.15.107:44558) 2024-03-25T19:39:19.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.780+0000 7fbd1ccf3640 1 -- 172.21.15.107:0/937824501 learned_addr learned my addr 172.21.15.107:0/937824501 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:19.781 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.781+0000 7fbd14ff9640 10 client.?.objecter ms_handle_connect 0x7fbd1814be90 2024-03-25T19:39:19.781 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.781+0000 7fbd14ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:19.786 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.787+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719692424 0 0) 0x7fbd181384b0 con 0x7fbd1814be90 2024-03-25T19:39:19.786 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.787+0000 7fbd14ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:19.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.787+0000 7fbd14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:19.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.787+0000 7fbd14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:19.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.787+0000 7fbd14ff9640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:39:19.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.787+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd04004eb0 con 0x7fbd1814be90 2024-03-25T19:39:19.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.792+0000 7fbd14ff9640 10 client.?.objecter ms_handle_connect 0x7fbd180b1730 2024-03-25T19:39:19.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.792+0000 7fbd14ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:19.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.800+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 231223689 0 0) 0x7fbd04004eb0 con 0x7fbd1814be90 2024-03-25T19:39:19.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.800+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd181384b0 con 0x7fbd1814be90 2024-03-25T19:39:19.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.801+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309635963 0 0) 0x7fbd181327e0 con 0x7fbd180b1730 2024-03-25T19:39:19.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.801+0000 7fbd14ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:19.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.801+0000 7fbd14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:19.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.801+0000 7fbd14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:19.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.801+0000 7fbd14ff9640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:39:19.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.801+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd04004eb0 con 0x7fbd180b1730 2024-03-25T19:39:19.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.802+0000 7fbd14ff9640 10 client.?.objecter ms_handle_connect 0x7fbd180b4d60 2024-03-25T19:39:19.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.802+0000 7fbd14ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:19.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.809+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fbd10002060 con 0x7fbd1814be90 2024-03-25T19:39:19.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.809+0000 7fbd14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.809+0000 7fbd14ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.809+0000 7fbd14ff9640 10 monclient(hunting): dump: 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:19.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.809+0000 7fbd14ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:19.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.811+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903955492 0 0) 0x7fbd18137e70 con 0x7fbd180b4d60 2024-03-25T19:39:19.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.811+0000 7fbd14ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:19.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.811+0000 7fbd14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:19.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.811+0000 7fbd14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:19.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.811+0000 7fbd14ff9640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:39:19.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.811+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd181327e0 con 0x7fbd180b4d60 2024-03-25T19:39:19.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 429343704 0 0) 0x7fbd181384b0 con 0x7fbd1814be90 2024-03-25T19:39:19.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 >> v1:172.21.15.107:6790/0 conn(0x7fbd180b4d60 legacy=0x7fbd181319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:19.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 >> v1:172.21.15.195:6789/0 conn(0x7fbd180b1730 legacy=0x7fbd181dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd14ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd14ff9640 1 monclient: found mon.a 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd14ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd14ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18138000 con 0x7fbd1814be90 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd14ff9640 10 monclient: _finish_auth 0 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd14ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd1e756640 5 monclient: authenticate success, global_id 4380 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd1e756640 10 monclient: _renew_subs 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd1e756640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd1e756640 1 -- 172.21.15.107:0/937824501 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd18145330 con 0x7fbd1814be90 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd1e756640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd1e756640 10 monclient: _renew_subs 2024-03-25T19:39:19.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd1e756640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:39:19.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd1e756640 1 -- 172.21.15.107:0/937824501 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd18140560 con 0x7fbd1814be90 2024-03-25T19:39:19.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.823+0000 7fbd1e756640 1 librados: init done 2024-03-25T19:39:19.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.825+0000 7fbd1e756640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:19.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.825+0000 7fbd1e756640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:19.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.825+0000 7fbd1e756640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:39:19.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.825+0000 7fbd1e756640 1 -- 172.21.15.107:0/937824501 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd18140810 con 0x7fbd1814be90 2024-03-25T19:39:19.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.852+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd10004e10 con 0x7fbd1814be90 2024-03-25T19:39:19.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.852+0000 7fbd14ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:19.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.852+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fbd10004f80 con 0x7fbd1814be90 2024-03-25T19:39:19.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.852+0000 7fbd14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:19.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.852+0000 7fbd14ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:19.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.852+0000 7fbd14ff9640 10 monclient: dump: 2024-03-25T19:39:19.852 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:19.852 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:19.852 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:19.852 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:19.853 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:19.853 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:19.853 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:19.853 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:19.853 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:19.853 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:19.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.852+0000 7fbd167fc640 4 set_mon_vals no callback set 2024-03-25T19:39:19.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.853+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fbd1001af50 con 0x7fbd1814be90 2024-03-25T19:39:19.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.854+0000 7fbd17fff640 10 monclient: get_auth_request con 0x7fbd0406de40 auth_method 0 2024-03-25T19:39:19.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:19.854+0000 7fbd14ff9640 10 client.4380.objecter ms_handle_connect 0x7fbd0406de40 2024-03-25T19:39:20.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.134+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fbd10054a20 con 0x7fbd1814be90 2024-03-25T19:39:20.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.134+0000 7fbd14ff9640 10 client.4380.objecter ms_dispatch 0x7fbd18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:20.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.134+0000 7fbd14ff9640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:20.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.134+0000 7fbd14ff9640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:20.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.134+0000 7fbd14ff9640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-25T19:39:20.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.134+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd10050530 con 0x7fbd1814be90 2024-03-25T19:39:20.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.134+0000 7fbd14ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:20.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.134+0000 7fbd14ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:20.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.400+0000 7fbd1e756640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:20.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.400+0000 7fbd1e756640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:20.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.400+0000 7fbd1e756640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:39:20.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.400+0000 7fbd1e756640 1 -- 172.21.15.107:0/937824501 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd180b22c0 con 0x7fbd1814be90 2024-03-25T19:39:20.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.509+0000 7fbd14ff9640 1 -- 172.21.15.107:0/937824501 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fbd10052460 con 0x7fbd1814be90 2024-03-25T19:39:20.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.509+0000 7fbd14ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:20.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.509+0000 7fbd14ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:20.510 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:39:20.510 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:20.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 10 librados: watch_flush enter 2024-03-25T19:39:20.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 10 librados: watch_flush exit 2024-03-25T19:39:20.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-25T19:39:20.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 10 client.4380.objecter successfully canceled tick 2024-03-25T19:39:20.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 1 -- 172.21.15.107:0/937824501 >> v1:172.21.15.195:6800/2984408421 conn(0x7fbd0406de40 legacy=0x7fbd04070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:20.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 10 monclient: shutdown 2024-03-25T19:39:20.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:20.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 1 -- 172.21.15.107:0/937824501 >> v1:172.21.15.107:6789/0 conn(0x7fbd1814be90 legacy=0x7fbd181320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:20.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 1 -- 172.21.15.107:0/937824501 shutdown_connections 2024-03-25T19:39:20.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 1 -- 172.21.15.107:0/937824501 >> 172.21.15.107:0/937824501 conn(0x7fbd180617a0 msgr2=0x7fbd1813c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:20.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 1 -- 172.21.15.107:0/937824501 shutdown_connections 2024-03-25T19:39:20.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 1 -- 172.21.15.107:0/937824501 wait complete. 2024-03-25T19:39:20.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:20.512+0000 7fbd1e756640 1 librados: shutdown 2024-03-25T19:39:20.524 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:24.849 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:03:55.165 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:24.858 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:03:55.174 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:24.907 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:03:55.224 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:24.911 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:03:55.227 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:24.912 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:03:55.199 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:24.916 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:03:55.203 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:24.924 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:03:55.240 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:24.955 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:03:55.242 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:24.964 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:03:55.281 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:27.330 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:27.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.438+0000 7f244c19d640 10 monclient: get_monmap_and_config 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.438+0000 7f244c19d640 10 monclient: build_initial_monmap 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.438+0000 7f244c19d640 10 monclient: monmap: 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:39:27.439192+0000 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:39:27.439192+0000 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:39:27.438 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:27.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.439+0000 7f244c19d640 1 Processor -- start 2024-03-25T19:39:27.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.439+0000 7f244c19d640 1 -- start start 2024-03-25T19:39:27.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.439+0000 7f244c19d640 10 monclient: init 2024-03-25T19:39:27.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.439+0000 7f244c19d640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:27.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.440+0000 7f244c19d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:27.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.440+0000 7f244c19d640 10 monclient(hunting): picked mon.noname-a con 0x7f24440a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:27.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.440+0000 7f244c19d640 10 monclient(hunting): picked mon.noname-b con 0x7f244413f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:27.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.440+0000 7f244c19d640 10 monclient(hunting): picked mon.noname-c con 0x7f24441430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:27.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.440+0000 7f244c19d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2444142520 con 0x7f24440a98f0 2024-03-25T19:39:27.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.440+0000 7f244c19d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24440b28a0 con 0x7f24441430c0 2024-03-25T19:39:27.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.440+0000 7f244c19d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24440b2a30 con 0x7f244413f710 2024-03-25T19:39:27.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.440+0000 7f244c19d640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:27.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.440+0000 7f244c19d640 10 monclient(hunting): authenticate will time out at 4759.780868s 2024-03-25T19:39:27.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.443+0000 7f2449f39640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f24440a98f0 0x7f24440a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40238/0 (socket says 172.21.15.107:40238) 2024-03-25T19:39:27.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.443+0000 7f2449f39640 1 -- 172.21.15.107:0/3074971719 learned_addr learned my addr 172.21.15.107:0/3074971719 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:27.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.458+0000 7f2448f37640 1 -- 172.21.15.107:0/3074971719 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072716803 0 0) 0x7f24440b2a30 con 0x7f244413f710 2024-03-25T19:39:27.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.458+0000 7f2448f37640 10 monclient(hunting): _negotiate 2024-03-25T19:39:27.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.458+0000 7f2448f37640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:27.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.458+0000 7f2448f37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:27.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.458+0000 7f2448f37640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:39:27.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.458+0000 7f2448f37640 1 -- 172.21.15.107:0/3074971719 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2430003520 con 0x7f244413f710 2024-03-25T19:39:27.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.470+0000 7f2448f37640 1 -- 172.21.15.107:0/3074971719 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458890592 0 0) 0x7f2444142520 con 0x7f24440a98f0 2024-03-25T19:39:27.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.470+0000 7f2448f37640 10 monclient(hunting): _negotiate 2024-03-25T19:39:27.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.470+0000 7f2448f37640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:27.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.470+0000 7f2448f37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:27.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.470+0000 7f2448f37640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:39:27.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.470+0000 7f2448f37640 1 -- 172.21.15.107:0/3074971719 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24440b2a30 con 0x7f24440a98f0 2024-03-25T19:39:27.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 1 -- 172.21.15.107:0/3074971719 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735623965 0 0) 0x7f2430003520 con 0x7f244413f710 2024-03-25T19:39:27.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 1 -- 172.21.15.107:0/3074971719 >> v1:172.21.15.195:6789/0 conn(0x7f24441430c0 legacy=0x7f244414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:27.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 1 -- 172.21.15.107:0/3074971719 >> v1:172.21.15.107:6789/0 conn(0x7f24440a98f0 legacy=0x7f24440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:27.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 10 monclient: _finish_hunting 0 2024-03-25T19:39:27.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 1 monclient: found mon.noname-b 2024-03-25T19:39:27.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:27.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:39:27.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 1 -- 172.21.15.107:0/3074971719 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24440b44c0 con 0x7f244413f710 2024-03-25T19:39:27.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 10 monclient: _finish_auth 0 2024-03-25T19:39:27.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 10 monclient: _check_auth_tickets 2024-03-25T19:39:27.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 1 -- 172.21.15.107:0/3074971719 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2434001690 con 0x7f244413f710 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f244c19d640 5 monclient: authenticate success, global_id 4385 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f244c19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f2448f37640 10 monclient: dump: 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:27.475 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:27.476 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:27.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.473+0000 7f244c19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:27.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.490+0000 7f2448f37640 1 -- 172.21.15.107:0/3074971719 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24340017f0 con 0x7f244413f710 2024-03-25T19:39:27.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.490+0000 7f2448f37640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:27.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.490+0000 7f2448f37640 1 -- 172.21.15.107:0/3074971719 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2434001c00 con 0x7f244413f710 2024-03-25T19:39:27.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.490+0000 7f2448f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:27.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.490+0000 7f2448f37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:27.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.490+0000 7f2448f37640 10 monclient: dump: 2024-03-25T19:39:27.490 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:27.490 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:27.491 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:27.491 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:27.491 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:27.491 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:27.491 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:27.491 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:27.491 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:27.491 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:27.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.490+0000 7f244c19d640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:27.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.490+0000 7f244c19d640 4 set_mon_vals no callback set 2024-03-25T19:39:27.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.491+0000 7f244c19d640 10 monclient: shutdown 2024-03-25T19:39:27.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.491+0000 7f244c19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:27.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.491+0000 7f244c19d640 1 -- 172.21.15.107:0/3074971719 >> v1:172.21.15.107:6790/0 conn(0x7f244413f710 legacy=0x7f24441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:27.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.491+0000 7f244c19d640 1 -- 172.21.15.107:0/3074971719 shutdown_connections 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.491+0000 7f244c19d640 1 -- 172.21.15.107:0/3074971719 >> 172.21.15.107:0/3074971719 conn(0x7f24440b7ea0 msgr2=0x7f24440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.491+0000 7f244c19d640 1 -- 172.21.15.107:0/3074971719 shutdown_connections 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.492+0000 7f244c19d640 1 -- 172.21.15.107:0/3074971719 wait complete. 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.492+0000 7f244c19d640 10 monclient: build_initial_monmap 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.492+0000 7f244c19d640 10 monclient: monmap: 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:39:27.494 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 1 librados: starting msgr at 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 1 librados: starting objecter 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 1 Processor -- start 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 1 -- start start 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 1 librados: setting wanted keys 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 1 librados: calling monclient init 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 10 monclient: init 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 10 monclient(hunting): picked mon.noname-a con 0x7f24440a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 10 monclient(hunting): picked mon.noname-b con 0x7f244413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 10 monclient(hunting): picked mon.noname-c con 0x7f24441430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24440ab8b0 con 0x7f24440a98f0 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24440abb90 con 0x7f244413f710 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2444137870 con 0x7f24441430c0 2024-03-25T19:39:27.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:27.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.493+0000 7f244c19d640 10 monclient(hunting): authenticate will time out at 4759.834610s 2024-03-25T19:39:27.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.503+0000 7f2449738640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f244413f710 0x7f24441df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:59418/0 (socket says 172.21.15.107:59418) 2024-03-25T19:39:27.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.503+0000 7f2449738640 1 -- 172.21.15.107:0/630648324 learned_addr learned my addr 172.21.15.107:0/630648324 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:27.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.503+0000 7f243a7fc640 10 client.?.objecter ms_handle_connect 0x7f244413f710 2024-03-25T19:39:27.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.503+0000 7f243a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:27.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.510+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224663523 0 0) 0x7f24440abb90 con 0x7f244413f710 2024-03-25T19:39:27.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.510+0000 7f243a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:27.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.510+0000 7f243a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:27.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.510+0000 7f243a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:27.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.510+0000 7f243a7fc640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:39:27.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.510+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2428004eb0 con 0x7f244413f710 2024-03-25T19:39:27.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.518+0000 7f243a7fc640 10 client.?.objecter ms_handle_connect 0x7f24441430c0 2024-03-25T19:39:27.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.518+0000 7f243a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:27.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.518+0000 7f243a7fc640 10 client.?.objecter ms_handle_connect 0x7f24440a98f0 2024-03-25T19:39:27.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.518+0000 7f243a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:27.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.521+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789762819 0 0) 0x7f24440ab8b0 con 0x7f24440a98f0 2024-03-25T19:39:27.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.521+0000 7f243a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:27.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.521+0000 7f243a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:27.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.521+0000 7f243a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:27.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.521+0000 7f243a7fc640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:39:27.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.521+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24440abb90 con 0x7f24440a98f0 2024-03-25T19:39:27.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.527+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2204949017 0 0) 0x7f2428004eb0 con 0x7f244413f710 2024-03-25T19:39:27.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.527+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24440ab8b0 con 0x7f244413f710 2024-03-25T19:39:27.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.527+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2434001690 con 0x7f244413f710 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.527+0000 7f243a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.527+0000 7f243a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.527+0000 7f243a7fc640 10 monclient(hunting): dump: 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:27.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.527+0000 7f243a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:27.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 763089358 0 0) 0x7f24440ab8b0 con 0x7f244413f710 2024-03-25T19:39:27.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 >> v1:172.21.15.107:6790/0 conn(0x7f24441430c0 legacy=0x7f24441e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:27.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 >> v1:172.21.15.107:6789/0 conn(0x7f24440a98f0 legacy=0x7f24441350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:27.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f243a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:27.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f243a7fc640 1 monclient: found mon.b 2024-03-25T19:39:27.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f243a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f243a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24441389e0 con 0x7f244413f710 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f243a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f243a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f243a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f244c19d640 5 monclient: authenticate success, global_id 4378 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f244c19d640 10 monclient: _renew_subs 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f244c19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f244c19d640 1 -- 172.21.15.107:0/630648324 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2444136d90 con 0x7f244413f710 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f244c19d640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f244c19d640 10 monclient: _renew_subs 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f244c19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f244c19d640 1 -- 172.21.15.107:0/630648324 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24441384c0 con 0x7f244413f710 2024-03-25T19:39:27.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.537+0000 7f244c19d640 1 librados: init done 2024-03-25T19:39:27.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.538+0000 7f244c19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:27.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.538+0000 7f244c19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:27.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.538+0000 7f244c19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:27.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.538+0000 7f244c19d640 1 -- 172.21.15.107:0/630648324 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24440627b0 con 0x7f244413f710 2024-03-25T19:39:27.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.543+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24340038a0 con 0x7f244413f710 2024-03-25T19:39:27.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.543+0000 7f243a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:27.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.543+0000 7f243bfff640 4 set_mon_vals no callback set 2024-03-25T19:39:27.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.819+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2434003c70 con 0x7f244413f710 2024-03-25T19:39:27.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.819+0000 7f243a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:27.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.819+0000 7f243a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:27.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.819+0000 7f243a7fc640 10 monclient: dump: 2024-03-25T19:39:27.819 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:27.819 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:27.819 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:27.819 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:27.819 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:27.819 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:27.820 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:27.820 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:27.820 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:27.820 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:27.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.820+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f2434003e50 con 0x7f244413f710 2024-03-25T19:39:27.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.821+0000 7f2449f39640 10 monclient: get_auth_request con 0x7f242806dda0 auth_method 0 2024-03-25T19:39:27.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.821+0000 7f243a7fc640 10 client.4378.objecter ms_handle_connect 0x7f242806dda0 2024-03-25T19:39:27.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.823+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f24340549f0 con 0x7f244413f710 2024-03-25T19:39:27.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.823+0000 7f243a7fc640 10 client.4378.objecter ms_dispatch 0x7f2444000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:27.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.823+0000 7f243a7fc640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:27.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.823+0000 7f243a7fc640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:27.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.823+0000 7f243a7fc640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T19:39:27.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.827+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2434050500 con 0x7f244413f710 2024-03-25T19:39:27.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.827+0000 7f243a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:27.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:27.827+0000 7f243a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:28.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.074+0000 7f244c19d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:28.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.074+0000 7f244c19d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:28.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.074+0000 7f244c19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:28.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.074+0000 7f244c19d640 1 -- 172.21.15.107:0/630648324 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2444138b70 con 0x7f244413f710 2024-03-25T19:39:28.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.099+0000 7f243a7fc640 1 -- 172.21.15.107:0/630648324 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f2434052430 con 0x7f244413f710 2024-03-25T19:39:28.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.099+0000 7f243a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:28.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.099+0000 7f243a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:28.099 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:39:28.099 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:28.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 10 librados: watch_flush enter 2024-03-25T19:39:28.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 10 librados: watch_flush exit 2024-03-25T19:39:28.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T19:39:28.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 10 client.4378.objecter successfully canceled tick 2024-03-25T19:39:28.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 1 -- 172.21.15.107:0/630648324 >> v1:172.21.15.195:6800/2984408421 conn(0x7f242806dda0 legacy=0x7f2428070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:28.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 10 monclient: shutdown 2024-03-25T19:39:28.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:28.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 1 -- 172.21.15.107:0/630648324 >> v1:172.21.15.195:6789/0 conn(0x7f244413f710 legacy=0x7f24441df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:28.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 1 -- 172.21.15.107:0/630648324 shutdown_connections 2024-03-25T19:39:28.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 1 -- 172.21.15.107:0/630648324 >> 172.21.15.107:0/630648324 conn(0x7f24440b7ea0 msgr2=0x7f2444074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:28.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 1 -- 172.21.15.107:0/630648324 shutdown_connections 2024-03-25T19:39:28.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 1 -- 172.21.15.107:0/630648324 wait complete. 2024-03-25T19:39:28.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:28.102+0000 7f244c19d640 1 librados: shutdown 2024-03-25T19:39:28.113 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:34.920 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:35.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.039+0000 7fdb7389a640 10 monclient: get_monmap_and_config 2024-03-25T19:39:35.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.039+0000 7fdb7389a640 10 monclient: build_initial_monmap 2024-03-25T19:39:35.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.039+0000 7fdb7389a640 10 monclient: monmap: 2024-03-25T19:39:35.039 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:35.039 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:35.039 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:39:35.040027+0000 2024-03-25T19:39:35.039 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:39:35.040027+0000 2024-03-25T19:39:35.039 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:35.040 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:35.040 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:35.040 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:39:35.040 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:39:35.040 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:35.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.039+0000 7fdb7389a640 1 Processor -- start 2024-03-25T19:39:35.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.039+0000 7fdb7389a640 1 -- start start 2024-03-25T19:39:35.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.039+0000 7fdb7389a640 10 monclient: init 2024-03-25T19:39:35.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.040+0000 7fdb7389a640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:35.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.040+0000 7fdb7389a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:35.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.040+0000 7fdb7389a640 10 monclient(hunting): picked mon.noname-a con 0x7fdb6c0a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:35.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.040+0000 7fdb7389a640 10 monclient(hunting): picked mon.noname-b con 0x7fdb6c13f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:35.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.040+0000 7fdb7389a640 10 monclient(hunting): picked mon.noname-c con 0x7fdb6c1430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:35.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.040+0000 7fdb7389a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c142520 con 0x7fdb6c0a98f0 2024-03-25T19:39:35.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.040+0000 7fdb7389a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c0b28a0 con 0x7fdb6c1430c0 2024-03-25T19:39:35.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.040+0000 7fdb7389a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c0b2a30 con 0x7fdb6c13f710 2024-03-25T19:39:35.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.040+0000 7fdb7389a640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:35.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.040+0000 7fdb7389a640 10 monclient(hunting): authenticate will time out at 4767.381333s 2024-03-25T19:39:35.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.047+0000 7fdb71636640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdb6c0a98f0 0x7fdb6c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39282/0 (socket says 172.21.15.107:39282) 2024-03-25T19:39:35.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.047+0000 7fdb71636640 1 -- 172.21.15.107:0/634121364 learned_addr learned my addr 172.21.15.107:0/634121364 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:35.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.056+0000 7fdb63fff640 1 -- 172.21.15.107:0/634121364 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882508510 0 0) 0x7fdb6c142520 con 0x7fdb6c0a98f0 2024-03-25T19:39:35.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.056+0000 7fdb63fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:35.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.056+0000 7fdb63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:35.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.056+0000 7fdb63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:35.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.056+0000 7fdb63fff640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:39:35.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.056+0000 7fdb63fff640 1 -- 172.21.15.107:0/634121364 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb58003520 con 0x7fdb6c0a98f0 2024-03-25T19:39:35.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 1 -- 172.21.15.107:0/634121364 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1573005480 0 0) 0x7fdb58003520 con 0x7fdb6c0a98f0 2024-03-25T19:39:35.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 1 -- 172.21.15.107:0/634121364 >> v1:172.21.15.107:6790/0 conn(0x7fdb6c13f710 legacy=0x7fdb6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:35.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 1 -- 172.21.15.107:0/634121364 >> v1:172.21.15.195:6789/0 conn(0x7fdb6c1430c0 legacy=0x7fdb6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:35.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 10 monclient: _finish_hunting 0 2024-03-25T19:39:35.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 1 monclient: found mon.noname-a 2024-03-25T19:39:35.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:35.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:39:35.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 1 -- 172.21.15.107:0/634121364 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c0b44c0 con 0x7fdb6c0a98f0 2024-03-25T19:39:35.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 10 monclient: _finish_auth 0 2024-03-25T19:39:35.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 10 monclient: _check_auth_tickets 2024-03-25T19:39:35.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb7389a640 5 monclient: authenticate success, global_id 4389 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 1 -- 172.21.15.107:0/634121364 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fdb68001690 con 0x7fdb6c0a98f0 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb63fff640 10 monclient: dump: 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:35.069 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:35.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.068+0000 7fdb7389a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:35.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.076+0000 7fdb63fff640 1 -- 172.21.15.107:0/634121364 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb680017f0 con 0x7fdb6c0a98f0 2024-03-25T19:39:35.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.076+0000 7fdb63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:35.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.076+0000 7fdb63fff640 1 -- 172.21.15.107:0/634121364 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fdb68001c00 con 0x7fdb6c0a98f0 2024-03-25T19:39:35.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.076+0000 7fdb7389a640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:35.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.076+0000 7fdb7389a640 4 set_mon_vals no callback set 2024-03-25T19:39:35.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.076+0000 7fdb63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:35.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.076+0000 7fdb63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:35.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.076+0000 7fdb63fff640 10 monclient: dump: 2024-03-25T19:39:35.076 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:35.076 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:35.077 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:35.077 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:35.077 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:35.077 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:35.077 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:35.077 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:35.077 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:35.077 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:35.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.077+0000 7fdb7389a640 10 monclient: shutdown 2024-03-25T19:39:35.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.077+0000 7fdb7389a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:35.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.077+0000 7fdb7389a640 1 -- 172.21.15.107:0/634121364 >> v1:172.21.15.107:6789/0 conn(0x7fdb6c0a98f0 legacy=0x7fdb6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:35.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.077+0000 7fdb7389a640 1 -- 172.21.15.107:0/634121364 shutdown_connections 2024-03-25T19:39:35.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.077+0000 7fdb7389a640 1 -- 172.21.15.107:0/634121364 >> 172.21.15.107:0/634121364 conn(0x7fdb6c0b7ea0 msgr2=0x7fdb6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.077+0000 7fdb7389a640 1 -- 172.21.15.107:0/634121364 shutdown_connections 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.077+0000 7fdb7389a640 1 -- 172.21.15.107:0/634121364 wait complete. 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.078+0000 7fdb7389a640 10 monclient: build_initial_monmap 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.078+0000 7fdb7389a640 10 monclient: monmap: 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.078+0000 7fdb7389a640 1 librados: starting msgr at 2024-03-25T19:39:35.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.078+0000 7fdb7389a640 1 librados: starting objecter 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.078+0000 7fdb7389a640 1 Processor -- start 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.078+0000 7fdb7389a640 1 -- start start 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.078+0000 7fdb7389a640 1 librados: setting wanted keys 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.078+0000 7fdb7389a640 1 librados: calling monclient init 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.078+0000 7fdb7389a640 10 monclient: init 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.079+0000 7fdb7389a640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.079+0000 7fdb7389a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.079+0000 7fdb7389a640 10 monclient(hunting): picked mon.noname-a con 0x7fdb6c0a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.079+0000 7fdb7389a640 10 monclient(hunting): picked mon.noname-b con 0x7fdb6c13f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.079+0000 7fdb7389a640 10 monclient(hunting): picked mon.noname-c con 0x7fdb6c1430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.079+0000 7fdb7389a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c135ac0 con 0x7fdb6c0a98f0 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.079+0000 7fdb7389a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c137030 con 0x7fdb6c13f710 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.079+0000 7fdb7389a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c1372a0 con 0x7fdb6c1430c0 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.079+0000 7fdb7389a640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:35.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.079+0000 7fdb7389a640 10 monclient(hunting): authenticate will time out at 4767.420329s 2024-03-25T19:39:35.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.098+0000 7fdb70e35640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fdb6c13f710 0x7fdb6c1379a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:55468/0 (socket says 172.21.15.107:55468) 2024-03-25T19:39:35.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.098+0000 7fdb70e35640 1 -- 172.21.15.107:0/1622479835 learned_addr learned my addr 172.21.15.107:0/1622479835 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:35.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.099+0000 7fdb61ffb640 10 client.?.objecter ms_handle_connect 0x7fdb6c13f710 2024-03-25T19:39:35.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.099+0000 7fdb61ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:35.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.112+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906354449 0 0) 0x7fdb6c137030 con 0x7fdb6c13f710 2024-03-25T19:39:35.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.112+0000 7fdb61ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:35.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.112+0000 7fdb61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:35.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.112+0000 7fdb61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:35.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.112+0000 7fdb61ffb640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:39:35.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.112+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb50004eb0 con 0x7fdb6c13f710 2024-03-25T19:39:35.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.112+0000 7fdb61ffb640 10 client.?.objecter ms_handle_connect 0x7fdb6c0a98f0 2024-03-25T19:39:35.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.112+0000 7fdb61ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:35.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.114+0000 7fdb61ffb640 10 client.?.objecter ms_handle_connect 0x7fdb6c1430c0 2024-03-25T19:39:35.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.114+0000 7fdb61ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:35.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.125+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826107079 0 0) 0x7fdb6c1372a0 con 0x7fdb6c1430c0 2024-03-25T19:39:35.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.125+0000 7fdb61ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:35.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.125+0000 7fdb61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:35.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.125+0000 7fdb61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:35.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.125+0000 7fdb61ffb640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:39:35.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.125+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb6c137030 con 0x7fdb6c1430c0 2024-03-25T19:39:35.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.129+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687286856 0 0) 0x7fdb6c135ac0 con 0x7fdb6c0a98f0 2024-03-25T19:39:35.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.129+0000 7fdb61ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:35.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.129+0000 7fdb61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:35.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.129+0000 7fdb61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:35.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.129+0000 7fdb61ffb640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:39:35.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.129+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb6c1372a0 con 0x7fdb6c0a98f0 2024-03-25T19:39:35.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.142+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4096829747 0 0) 0x7fdb50004eb0 con 0x7fdb6c13f710 2024-03-25T19:39:35.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.142+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb6c135ac0 con 0x7fdb6c13f710 2024-03-25T19:39:35.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.142+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fdb5c001a30 con 0x7fdb6c13f710 2024-03-25T19:39:35.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.142+0000 7fdb61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:35.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.142+0000 7fdb61ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:35.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.142+0000 7fdb61ffb640 10 monclient(hunting): dump: 2024-03-25T19:39:35.142 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:35.142 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:35.142 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:35.142 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:35.143 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:35.143 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:35.143 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:35.143 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:35.143 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:35.143 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:35.143 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.142+0000 7fdb61ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.150+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 547448535 0 0) 0x7fdb6c137030 con 0x7fdb6c1430c0 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.150+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb50004eb0 con 0x7fdb6c1430c0 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.150+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fdb64001b10 con 0x7fdb6c1430c0 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.150+0000 7fdb61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.150+0000 7fdb61ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.150+0000 7fdb61ffb640 10 monclient(hunting): dump: 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:35.150 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:35.151 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:35.151 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:35.151 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:35.151 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:35.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3594278603 0 0) 0x7fdb6c135ac0 con 0x7fdb6c13f710 2024-03-25T19:39:35.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 >> v1:172.21.15.107:6790/0 conn(0x7fdb6c1430c0 legacy=0x7fdb6c1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 >> v1:172.21.15.107:6789/0 conn(0x7fdb6c0a98f0 legacy=0x7fdb6c133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb61ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb61ffb640 1 monclient: found mon.b 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb61ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c137580 con 0x7fdb6c13f710 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb61ffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb61ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb7389a640 5 monclient: authenticate success, global_id 4384 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb7389a640 10 monclient: _renew_subs 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb7389a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb7389a640 1 -- 172.21.15.107:0/1622479835 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6c1363b0 con 0x7fdb6c13f710 2024-03-25T19:39:35.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb7389a640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:35.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb7389a640 10 monclient: _renew_subs 2024-03-25T19:39:35.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb7389a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:35.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb7389a640 1 -- 172.21.15.107:0/1622479835 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6c1353a0 con 0x7fdb6c13f710 2024-03-25T19:39:35.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.158+0000 7fdb7389a640 1 librados: init done 2024-03-25T19:39:35.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.159+0000 7fdb7389a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:35.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.160+0000 7fdb7389a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:35.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.160+0000 7fdb7389a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:35.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.160+0000 7fdb7389a640 1 -- 172.21.15.107:0/1622479835 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6c130d90 con 0x7fdb6c13f710 2024-03-25T19:39:35.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.438+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb5c001c70 con 0x7fdb6c13f710 2024-03-25T19:39:35.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.438+0000 7fdb61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:35.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.438+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fdb5c0038c0 con 0x7fdb6c13f710 2024-03-25T19:39:35.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.438+0000 7fdb61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:35.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.438+0000 7fdb61ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.438+0000 7fdb61ffb640 10 monclient: dump: 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:35.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.438+0000 7fdb637fe640 4 set_mon_vals no callback set 2024-03-25T19:39:35.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.439+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fdb5c003aa0 con 0x7fdb6c13f710 2024-03-25T19:39:35.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.440+0000 7fdb71636640 10 monclient: get_auth_request con 0x7fdb5006df70 auth_method 0 2024-03-25T19:39:35.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.440+0000 7fdb61ffb640 10 client.4384.objecter ms_handle_connect 0x7fdb5006df70 2024-03-25T19:39:35.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.444+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fdb5c054660 con 0x7fdb6c13f710 2024-03-25T19:39:35.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.444+0000 7fdb61ffb640 10 client.4384.objecter ms_dispatch 0x7fdb6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:35.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.444+0000 7fdb61ffb640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:35.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.444+0000 7fdb61ffb640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:35.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.444+0000 7fdb61ffb640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T19:39:35.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.445+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb5c050170 con 0x7fdb6c13f710 2024-03-25T19:39:35.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.445+0000 7fdb61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:35.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.445+0000 7fdb61ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:35.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.670+0000 7fdb7389a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:35.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.670+0000 7fdb7389a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:35.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.670+0000 7fdb7389a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:35.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.670+0000 7fdb7389a640 1 -- 172.21.15.107:0/1622479835 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb6c074cc0 con 0x7fdb6c13f710 2024-03-25T19:39:35.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.704+0000 7fdb61ffb640 1 -- 172.21.15.107:0/1622479835 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fdb5c0520a0 con 0x7fdb6c13f710 2024-03-25T19:39:35.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.704+0000 7fdb61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:35.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.704+0000 7fdb61ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:35.705 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:39:35.705 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.706+0000 7fdb7389a640 10 librados: watch_flush enter 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.706+0000 7fdb7389a640 10 librados: watch_flush exit 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.706+0000 7fdb7389a640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.706+0000 7fdb7389a640 10 client.4384.objecter successfully canceled tick 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.707+0000 7fdb7389a640 1 -- 172.21.15.107:0/1622479835 >> v1:172.21.15.195:6800/2984408421 conn(0x7fdb5006df70 legacy=0x7fdb50070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.707+0000 7fdb7389a640 10 monclient: shutdown 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.707+0000 7fdb7389a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.707+0000 7fdb7389a640 1 -- 172.21.15.107:0/1622479835 >> v1:172.21.15.195:6789/0 conn(0x7fdb6c13f710 legacy=0x7fdb6c1379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.707+0000 7fdb7389a640 1 -- 172.21.15.107:0/1622479835 shutdown_connections 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.707+0000 7fdb7389a640 1 -- 172.21.15.107:0/1622479835 >> 172.21.15.107:0/1622479835 conn(0x7fdb6c0b7ea0 msgr2=0x7fdb6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.707+0000 7fdb7389a640 1 -- 172.21.15.107:0/1622479835 shutdown_connections 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.707+0000 7fdb7389a640 1 -- 172.21.15.107:0/1622479835 wait complete. 2024-03-25T19:39:35.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:35.707+0000 7fdb7389a640 1 librados: shutdown 2024-03-25T19:39:35.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:42.524 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:42.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.643+0000 7f2acd29c640 10 monclient: get_monmap_and_config 2024-03-25T19:39:42.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.643+0000 7f2acd29c640 10 monclient: build_initial_monmap 2024-03-25T19:39:42.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.643+0000 7f2acd29c640 10 monclient: monmap: 2024-03-25T19:39:42.643 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:42.643 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:42.643 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:39:42.644618+0000 2024-03-25T19:39:42.643 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:39:42.644618+0000 2024-03-25T19:39:42.643 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:42.643 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:42.644 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:42.644 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:39:42.644 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:39:42.644 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:42.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.644+0000 7f2acd29c640 1 Processor -- start 2024-03-25T19:39:42.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.644+0000 7f2acd29c640 1 -- start start 2024-03-25T19:39:42.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.644+0000 7f2acd29c640 10 monclient: init 2024-03-25T19:39:42.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.645+0000 7f2acd29c640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:42.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.645+0000 7f2acd29c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:42.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.645+0000 7f2acd29c640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac806c380 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:42.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.645+0000 7f2acd29c640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac80b12d0 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:42.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.645+0000 7f2acd29c640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac80aedf0 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:42.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.645+0000 7f2acd29c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80b16f0 con 0x7f2ac80aedf0 2024-03-25T19:39:42.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.645+0000 7f2acd29c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80adfe0 con 0x7f2ac80b12d0 2024-03-25T19:39:42.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.645+0000 7f2acd29c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80add00 con 0x7f2ac806c380 2024-03-25T19:39:42.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.645+0000 7f2acd29c640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:42.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.645+0000 7f2acd29c640 10 monclient(hunting): authenticate will time out at 4774.985947s 2024-03-25T19:39:42.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.649+0000 7f2ac67fc640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2ac80b12d0 0x7f2ac80ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:50198/0 (socket says 172.21.15.107:50198) 2024-03-25T19:39:42.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.649+0000 7f2ac67fc640 1 -- 172.21.15.107:0/668745191 learned_addr learned my addr 172.21.15.107:0/668745191 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:42.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.657+0000 7f2ac5ffb640 1 -- 172.21.15.107:0/668745191 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322881740 0 0) 0x7f2ac80adfe0 con 0x7f2ac80b12d0 2024-03-25T19:39:42.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.657+0000 7f2ac5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:42.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.657+0000 7f2ac5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:42.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.657+0000 7f2ac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:42.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.657+0000 7f2ac5ffb640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:39:42.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.657+0000 7f2ac5ffb640 1 -- 172.21.15.107:0/668745191 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab0003520 con 0x7f2ac80b12d0 2024-03-25T19:39:42.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.664+0000 7f2ac5ffb640 1 -- 172.21.15.107:0/668745191 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390976608 0 0) 0x7f2ac80add00 con 0x7f2ac806c380 2024-03-25T19:39:42.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.664+0000 7f2ac5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:42.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.664+0000 7f2ac5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:42.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.664+0000 7f2ac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:42.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.664+0000 7f2ac5ffb640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:39:42.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.664+0000 7f2ac5ffb640 1 -- 172.21.15.107:0/668745191 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac80adfe0 con 0x7f2ac806c380 2024-03-25T19:39:42.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 1 -- 172.21.15.107:0/668745191 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4019511678 0 0) 0x7f2ab0003520 con 0x7f2ac80b12d0 2024-03-25T19:39:42.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 1 -- 172.21.15.107:0/668745191 >> v1:172.21.15.107:6790/0 conn(0x7f2ac806c380 legacy=0x7f2ac80b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:42.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 1 -- 172.21.15.107:0/668745191 >> v1:172.21.15.107:6789/0 conn(0x7f2ac80aedf0 legacy=0x7f2ac814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:42.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:42.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:39:42.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:42.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 1 -- 172.21.15.107:0/668745191 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80ae2c0 con 0x7f2ac80b12d0 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 1 -- 172.21.15.107:0/668745191 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2abc001690 con 0x7f2ac80b12d0 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2acd29c640 5 monclient: authenticate success, global_id 4387 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2acd29c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2ac5ffb640 10 monclient: dump: 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:42.669 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:42.670 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:42.670 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:42.670 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:42.670 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:42.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.668+0000 7f2acd29c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:42.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2ac5ffb640 1 -- 172.21.15.107:0/668745191 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2abc0017f0 con 0x7f2ac80b12d0 2024-03-25T19:39:42.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2ac5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:42.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2ac5ffb640 1 -- 172.21.15.107:0/668745191 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2abc001b80 con 0x7f2ac80b12d0 2024-03-25T19:39:42.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2acd29c640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:42.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2acd29c640 4 set_mon_vals no callback set 2024-03-25T19:39:42.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2ac5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:42.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2ac5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2ac5ffb640 10 monclient: dump: 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2acd29c640 10 monclient: shutdown 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2acd29c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:42.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2acd29c640 1 -- 172.21.15.107:0/668745191 >> v1:172.21.15.195:6789/0 conn(0x7f2ac80b12d0 legacy=0x7f2ac80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:42.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2acd29c640 1 -- 172.21.15.107:0/668745191 shutdown_connections 2024-03-25T19:39:42.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.687+0000 7f2acd29c640 1 -- 172.21.15.107:0/668745191 >> 172.21.15.107:0/668745191 conn(0x7f2ac80b5d10 msgr2=0x7f2ac80b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:42.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.688+0000 7f2acd29c640 1 -- 172.21.15.107:0/668745191 shutdown_connections 2024-03-25T19:39:42.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.688+0000 7f2acd29c640 1 -- 172.21.15.107:0/668745191 wait complete. 2024-03-25T19:39:42.690 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.688+0000 7f2acd29c640 10 monclient: build_initial_monmap 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.688+0000 7f2acd29c640 10 monclient: monmap: 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.689+0000 7f2acd29c640 1 librados: starting msgr at 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.689+0000 7f2acd29c640 1 librados: starting objecter 2024-03-25T19:39:42.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.689+0000 7f2acd29c640 1 Processor -- start 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.689+0000 7f2acd29c640 1 -- start start 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.689+0000 7f2acd29c640 1 librados: setting wanted keys 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.689+0000 7f2acd29c640 1 librados: calling monclient init 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.689+0000 7f2acd29c640 10 monclient: init 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.689+0000 7f2acd29c640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.689+0000 7f2acd29c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.689+0000 7f2acd29c640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac806c380 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.690+0000 7f2acd29c640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac80aedf0 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.690+0000 7f2acd29c640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac80b12d0 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.690+0000 7f2acd29c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac806c8e0 con 0x7f2ac80aedf0 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.690+0000 7f2acd29c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac81db250 con 0x7f2ac806c380 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.690+0000 7f2acd29c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac81db9d0 con 0x7f2ac80b12d0 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.690+0000 7f2acd29c640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:42.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.690+0000 7f2acd29c640 10 monclient(hunting): authenticate will time out at 4775.030893s 2024-03-25T19:39:42.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.698+0000 7f2ac67fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2ac80aedf0 0x7f2ac81c07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35660/0 (socket says 172.21.15.107:35660) 2024-03-25T19:39:42.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.698+0000 7f2ac67fc640 1 -- 172.21.15.107:0/4152072714 learned_addr learned my addr 172.21.15.107:0/4152072714 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:42.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.699+0000 7f2aaf7fe640 10 client.?.objecter ms_handle_connect 0x7f2ac80aedf0 2024-03-25T19:39:42.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.699+0000 7f2aaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:42.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.702+0000 7f2aaf7fe640 10 client.?.objecter ms_handle_connect 0x7f2ac80b12d0 2024-03-25T19:39:42.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.702+0000 7f2aaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:42.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.705+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365365521 0 0) 0x7f2ac806c8e0 con 0x7f2ac80aedf0 2024-03-25T19:39:42.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.705+0000 7f2aaf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:42.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.705+0000 7f2aaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:42.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.705+0000 7f2aaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:42.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.705+0000 7f2aaf7fe640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:39:42.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.705+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab4004eb0 con 0x7f2ac80aedf0 2024-03-25T19:39:42.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.711+0000 7f2aaf7fe640 10 client.?.objecter ms_handle_connect 0x7f2ac806c380 2024-03-25T19:39:42.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.711+0000 7f2aaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:42.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674833803 0 0) 0x7f2ac81db9d0 con 0x7f2ac80b12d0 2024-03-25T19:39:42.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:42.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:42.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:42.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:39:42.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac806c8e0 con 0x7f2ac80b12d0 2024-03-25T19:39:42.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 748585292 0 0) 0x7f2ab4004eb0 con 0x7f2ac80aedf0 2024-03-25T19:39:42.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ac81db9d0 con 0x7f2ac80aedf0 2024-03-25T19:39:42.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2abc001b40 con 0x7f2ac80aedf0 2024-03-25T19:39:42.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:42.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 10 monclient(hunting): dump: 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:42.714 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.713+0000 7f2aaf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:42.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1492814157 0 0) 0x7f2ac81db9d0 con 0x7f2ac80aedf0 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 >> v1:172.21.15.107:6790/0 conn(0x7f2ac80b12d0 legacy=0x7f2ac81c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 >> v1:172.21.15.195:6789/0 conn(0x7f2ac806c380 legacy=0x7f2ac81dab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2aaf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2aaf7fe640 1 monclient: found mon.a 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2aaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2aaf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac81db4b0 con 0x7f2ac80aedf0 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2aaf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2aaf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2aaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2acd29c640 5 monclient: authenticate success, global_id 4398 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2acd29c640 10 monclient: _renew_subs 2024-03-25T19:39:42.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2acd29c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:39:42.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2acd29c640 1 -- 172.21.15.107:0/4152072714 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac81c1880 con 0x7f2ac80aedf0 2024-03-25T19:39:42.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2acd29c640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:42.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2acd29c640 10 monclient: _renew_subs 2024-03-25T19:39:42.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2acd29c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:39:42.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2acd29c640 1 -- 172.21.15.107:0/4152072714 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac81c10d0 con 0x7f2ac80aedf0 2024-03-25T19:39:42.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.720+0000 7f2acd29c640 1 librados: init done 2024-03-25T19:39:42.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.721+0000 7f2acd29c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:42.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.721+0000 7f2acd29c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:42.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.721+0000 7f2acd29c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:39:42.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:42.721+0000 7f2acd29c640 1 -- 172.21.15.107:0/4152072714 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac806d790 con 0x7f2ac80aedf0 2024-03-25T19:39:43.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.027+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2abc0011f0 con 0x7f2ac80aedf0 2024-03-25T19:39:43.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.027+0000 7f2aaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:43.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.027+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2abc004a90 con 0x7f2ac80aedf0 2024-03-25T19:39:43.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.027+0000 7f2aaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:43.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.027+0000 7f2aaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:43.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.027+0000 7f2aaf7fe640 10 monclient: dump: 2024-03-25T19:39:43.027 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:43.028 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:43.028 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:43.028 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:43.028 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:43.028 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:43.028 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:43.028 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:43.028 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:43.028 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:43.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.027+0000 7f2ac57fa640 4 set_mon_vals no callback set 2024-03-25T19:39:43.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.028+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f2abc01aa60 con 0x7f2ac80aedf0 2024-03-25T19:39:43.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.029+0000 7f2ac6ffd640 10 monclient: get_auth_request con 0x7f2ab406dd90 auth_method 0 2024-03-25T19:39:43.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.029+0000 7f2aaf7fe640 10 client.4398.objecter ms_handle_connect 0x7f2ab406dd90 2024-03-25T19:39:43.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.034+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f2abc054700 con 0x7f2ac80aedf0 2024-03-25T19:39:43.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.034+0000 7f2aaf7fe640 10 client.4398.objecter ms_dispatch 0x7f2ac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:43.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.034+0000 7f2aaf7fe640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:43.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.034+0000 7f2aaf7fe640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:43.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.034+0000 7f2aaf7fe640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-25T19:39:43.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.034+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2abc050040 con 0x7f2ac80aedf0 2024-03-25T19:39:43.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.034+0000 7f2aaf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:43.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.034+0000 7f2aaf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:43.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.276+0000 7f2acd29c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:43.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.276+0000 7f2acd29c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:43.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.276+0000 7f2acd29c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:39:43.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.276+0000 7f2acd29c640 1 -- 172.21.15.107:0/4152072714 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ac80627b0 con 0x7f2ac80aedf0 2024-03-25T19:39:43.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.302+0000 7f2aaf7fe640 1 -- 172.21.15.107:0/4152072714 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f2abc052140 con 0x7f2ac80aedf0 2024-03-25T19:39:43.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.302+0000 7f2aaf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:43.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.302+0000 7f2aaf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:43.302 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:39:43.302 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:43.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 10 librados: watch_flush enter 2024-03-25T19:39:43.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 10 librados: watch_flush exit 2024-03-25T19:39:43.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-25T19:39:43.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 10 client.4398.objecter successfully canceled tick 2024-03-25T19:39:43.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 1 -- 172.21.15.107:0/4152072714 >> v1:172.21.15.195:6800/2984408421 conn(0x7f2ab406dd90 legacy=0x7f2ab4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:43.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 10 monclient: shutdown 2024-03-25T19:39:43.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:43.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 1 -- 172.21.15.107:0/4152072714 >> v1:172.21.15.107:6789/0 conn(0x7f2ac80aedf0 legacy=0x7f2ac81c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:43.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 1 -- 172.21.15.107:0/4152072714 shutdown_connections 2024-03-25T19:39:43.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 1 -- 172.21.15.107:0/4152072714 >> 172.21.15.107:0/4152072714 conn(0x7f2ac80b5d10 msgr2=0x7f2ac81cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:43.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 1 -- 172.21.15.107:0/4152072714 shutdown_connections 2024-03-25T19:39:43.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 1 -- 172.21.15.107:0/4152072714 wait complete. 2024-03-25T19:39:43.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:43.304+0000 7f2acd29c640 1 librados: shutdown 2024-03-25T19:39:43.316 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:50.123 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:50.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.227+0000 7ff6527d6640 10 monclient: get_monmap_and_config 2024-03-25T19:39:50.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.227+0000 7ff6527d6640 10 monclient: build_initial_monmap 2024-03-25T19:39:50.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.228+0000 7ff6527d6640 10 monclient: monmap: 2024-03-25T19:39:50.228 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:50.228 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:50.228 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:39:50.228975+0000 2024-03-25T19:39:50.228 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:39:50.228975+0000 2024-03-25T19:39:50.228 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:50.228 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:50.228 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:50.228 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:39:50.228 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:39:50.228 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:50.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.228+0000 7ff6527d6640 1 Processor -- start 2024-03-25T19:39:50.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.228+0000 7ff6527d6640 1 -- start start 2024-03-25T19:39:50.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.228+0000 7ff6527d6640 10 monclient: init 2024-03-25T19:39:50.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.229+0000 7ff6527d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:50.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.229+0000 7ff6527d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:50.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.229+0000 7ff6527d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff64c061830 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:50.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.229+0000 7ff6527d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff64c1400e0 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:50.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.229+0000 7ff6527d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff64c1499d0 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:50.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.229+0000 7ff6527d6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c072600 con 0x7ff64c061830 2024-03-25T19:39:50.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.229+0000 7ff6527d6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c140c80 con 0x7ff64c1499d0 2024-03-25T19:39:50.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.229+0000 7ff6527d6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c072830 con 0x7ff64c1400e0 2024-03-25T19:39:50.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.229+0000 7ff6527d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:50.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.229+0000 7ff6527d6640 10 monclient(hunting): authenticate will time out at 4782.570383s 2024-03-25T19:39:50.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.238+0000 7ff64bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff64c061830 0x7ff64c13d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40446/0 (socket says 172.21.15.107:40446) 2024-03-25T19:39:50.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.238+0000 7ff64bfff640 1 -- 172.21.15.107:0/1655413805 learned_addr learned my addr 172.21.15.107:0/1655413805 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:50.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.256+0000 7ff64affd640 1 -- 172.21.15.107:0/1655413805 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016923751 0 0) 0x7ff64c072830 con 0x7ff64c1400e0 2024-03-25T19:39:50.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.256+0000 7ff64affd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:50.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.256+0000 7ff64affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:50.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.256+0000 7ff64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:50.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.256+0000 7ff64affd640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:39:50.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.256+0000 7ff64affd640 1 -- 172.21.15.107:0/1655413805 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff634003520 con 0x7ff64c1400e0 2024-03-25T19:39:50.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.256+0000 7ff64affd640 1 -- 172.21.15.107:0/1655413805 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288852221 0 0) 0x7ff64c072600 con 0x7ff64c061830 2024-03-25T19:39:50.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.257+0000 7ff64affd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:50.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.257+0000 7ff64affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:50.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.257+0000 7ff64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:50.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.257+0000 7ff64affd640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:39:50.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.257+0000 7ff64affd640 1 -- 172.21.15.107:0/1655413805 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff64c072830 con 0x7ff64c061830 2024-03-25T19:39:50.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 1 -- 172.21.15.107:0/1655413805 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3600019744 0 0) 0x7ff634003520 con 0x7ff64c1400e0 2024-03-25T19:39:50.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 1 -- 172.21.15.107:0/1655413805 >> v1:172.21.15.195:6789/0 conn(0x7ff64c1499d0 legacy=0x7ff64c14bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 1 -- 172.21.15.107:0/1655413805 >> v1:172.21.15.107:6789/0 conn(0x7ff64c061830 legacy=0x7ff64c13d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 1 monclient: found mon.noname-b 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 1 -- 172.21.15.107:0/1655413805 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c072b10 con 0x7ff64c1400e0 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 10 monclient: _finish_auth 0 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 1 -- 172.21.15.107:0/1655413805 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff640001690 con 0x7ff64c1400e0 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff64affd640 10 monclient: dump: 2024-03-25T19:39:50.469 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff6527d6640 5 monclient: authenticate success, global_id 4403 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.275+0000 7ff6527d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.297+0000 7ff64affd640 1 -- 172.21.15.107:0/1655413805 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6400017f0 con 0x7ff64c1400e0 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.297+0000 7ff64affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:50.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.297+0000 7ff64affd640 1 -- 172.21.15.107:0/1655413805 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff640001c00 con 0x7ff64c1400e0 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.297+0000 7ff6527d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.297+0000 7ff6527d6640 4 set_mon_vals no callback set 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.297+0000 7ff64affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.297+0000 7ff64affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.297+0000 7ff64affd640 10 monclient: dump: 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:50.471 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.298+0000 7ff6527d6640 10 monclient: shutdown 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.298+0000 7ff6527d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.298+0000 7ff6527d6640 1 -- 172.21.15.107:0/1655413805 >> v1:172.21.15.107:6790/0 conn(0x7ff64c1400e0 legacy=0x7ff64c140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.298+0000 7ff6527d6640 1 -- 172.21.15.107:0/1655413805 shutdown_connections 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.298+0000 7ff6527d6640 1 -- 172.21.15.107:0/1655413805 >> 172.21.15.107:0/1655413805 conn(0x7ff64c0b0ef0 msgr2=0x7ff64c0b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.298+0000 7ff6527d6640 1 -- 172.21.15.107:0/1655413805 shutdown_connections 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.298+0000 7ff6527d6640 1 -- 172.21.15.107:0/1655413805 wait complete. 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.299+0000 7ff6527d6640 10 monclient: build_initial_monmap 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.299+0000 7ff6527d6640 10 monclient: monmap: 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:50.472 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.299+0000 7ff6527d6640 1 librados: starting msgr at 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.299+0000 7ff6527d6640 1 librados: starting objecter 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.299+0000 7ff6527d6640 1 Processor -- start 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 1 -- start start 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 1 librados: setting wanted keys 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 1 librados: calling monclient init 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 10 monclient: init 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff64c061830 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff64c1400e0 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:50.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff64c1499d0 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c0ab020 con 0x7ff64c061830 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c0ab300 con 0x7ff64c1499d0 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c136bc0 con 0x7ff64c1400e0 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.300+0000 7ff6527d6640 10 monclient(hunting): authenticate will time out at 4782.641661s 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.315+0000 7ff650d73640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff64c1499d0 0x7ff64c1e09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:55040/0 (socket says 172.21.15.107:55040) 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.315+0000 7ff650d73640 1 -- 172.21.15.107:0/721865965 learned_addr learned my addr 172.21.15.107:0/721865965 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.316+0000 7ff648ff9640 10 client.?.objecter ms_handle_connect 0x7ff64c1499d0 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.316+0000 7ff648ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.319+0000 7ff648ff9640 10 client.?.objecter ms_handle_connect 0x7ff64c1400e0 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.319+0000 7ff648ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.320+0000 7ff648ff9640 10 client.?.objecter ms_handle_connect 0x7ff64c061830 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.320+0000 7ff648ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.321+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910427706 0 0) 0x7ff64c0ab300 con 0x7ff64c1499d0 2024-03-25T19:39:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.321+0000 7ff648ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.321+0000 7ff648ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.321+0000 7ff648ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.321+0000 7ff648ff9640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.321+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff638004eb0 con 0x7ff64c1499d0 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.328+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175826006 0 0) 0x7ff64c0ab020 con 0x7ff64c061830 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.328+0000 7ff648ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.328+0000 7ff648ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.328+0000 7ff648ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.328+0000 7ff648ff9640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.328+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff64c0ab300 con 0x7ff64c061830 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.330+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1751753889 0 0) 0x7ff638004eb0 con 0x7ff64c1499d0 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.330+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff64c0ab020 con 0x7ff64c1499d0 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.330+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff644001c90 con 0x7ff64c1499d0 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.330+0000 7ff648ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:50.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.330+0000 7ff648ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.330+0000 7ff648ff9640 10 monclient(hunting): dump: 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.330+0000 7ff648ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.335+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735635766 0 0) 0x7ff64c136bc0 con 0x7ff64c1400e0 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.335+0000 7ff648ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:50.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.335+0000 7ff648ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.335+0000 7ff648ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.335+0000 7ff648ff9640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.335+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff638004eb0 con 0x7ff64c1400e0 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.350+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2459249228 0 0) 0x7ff64c0ab020 con 0x7ff64c1499d0 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 >> v1:172.21.15.107:6790/0 conn(0x7ff64c1400e0 legacy=0x7ff64c133600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 >> v1:172.21.15.107:6789/0 conn(0x7ff64c061830 legacy=0x7ff64c1d50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff648ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff648ff9640 1 monclient: found mon.b 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff648ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff648ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c0aa040 con 0x7ff64c1499d0 2024-03-25T19:39:50.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff648ff9640 10 monclient: _finish_auth 0 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff648ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff648ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff6527d6640 5 monclient: authenticate success, global_id 4393 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff6527d6640 10 monclient: _renew_subs 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff6527d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff6527d6640 1 -- 172.21.15.107:0/721865965 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c1d57d0 con 0x7ff64c1499d0 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff6527d6640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff6527d6640 10 monclient: _renew_subs 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff6527d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff6527d6640 1 -- 172.21.15.107:0/721865965 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c133ef0 con 0x7ff64c1499d0 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.351+0000 7ff6527d6640 1 librados: init done 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.352+0000 7ff6527d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.352+0000 7ff6527d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:50.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.352+0000 7ff6527d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:50.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.352+0000 7ff6527d6640 1 -- 172.21.15.107:0/721865965 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64c0a9c80 con 0x7ff64c1499d0 2024-03-25T19:39:50.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.629+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff644003990 con 0x7ff64c1499d0 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.629+0000 7ff648ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.629+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff644003da0 con 0x7ff64c1499d0 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.629+0000 7ff648ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.629+0000 7ff648ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.629+0000 7ff648ff9640 10 monclient: dump: 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:50.630 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:50.631 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:50.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.629+0000 7ff64a7fc640 4 set_mon_vals no callback set 2024-03-25T19:39:50.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.630+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7ff644003fc0 con 0x7ff64c1499d0 2024-03-25T19:39:50.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.631+0000 7ff64bfff640 10 monclient: get_auth_request con 0x7ff63806de20 auth_method 0 2024-03-25T19:39:50.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.631+0000 7ff648ff9640 10 client.4393.objecter ms_handle_connect 0x7ff63806de20 2024-03-25T19:39:50.635 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.635+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7ff644054b00 con 0x7ff64c1499d0 2024-03-25T19:39:50.635 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.635+0000 7ff648ff9640 10 client.4393.objecter ms_dispatch 0x7ff64c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:50.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.635+0000 7ff648ff9640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:50.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.635+0000 7ff648ff9640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:50.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.635+0000 7ff648ff9640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-25T19:39:50.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.636+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff644050610 con 0x7ff64c1499d0 2024-03-25T19:39:50.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.636+0000 7ff648ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:50.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.636+0000 7ff648ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:50.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.881+0000 7ff6527d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:50.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.881+0000 7ff6527d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:50.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.881+0000 7ff6527d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:50.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.881+0000 7ff6527d6640 1 -- 172.21.15.107:0/721865965 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff64c0627b0 con 0x7ff64c1499d0 2024-03-25T19:39:50.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.909+0000 7ff648ff9640 1 -- 172.21.15.107:0/721865965 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7ff644052540 con 0x7ff64c1499d0 2024-03-25T19:39:50.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.909+0000 7ff648ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:50.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.909+0000 7ff648ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:50.909 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:39:50.909 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:50.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 10 librados: watch_flush enter 2024-03-25T19:39:50.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 10 librados: watch_flush exit 2024-03-25T19:39:50.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-25T19:39:50.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 10 client.4393.objecter successfully canceled tick 2024-03-25T19:39:50.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 1 -- 172.21.15.107:0/721865965 >> v1:172.21.15.195:6800/2984408421 conn(0x7ff63806de20 legacy=0x7ff6380702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:50.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 10 monclient: shutdown 2024-03-25T19:39:50.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:50.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 1 -- 172.21.15.107:0/721865965 >> v1:172.21.15.195:6789/0 conn(0x7ff64c1499d0 legacy=0x7ff64c1e09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:50.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 1 -- 172.21.15.107:0/721865965 shutdown_connections 2024-03-25T19:39:50.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 1 -- 172.21.15.107:0/721865965 >> 172.21.15.107:0/721865965 conn(0x7ff64c0b0ef0 msgr2=0x7ff64c13a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:50.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 1 -- 172.21.15.107:0/721865965 shutdown_connections 2024-03-25T19:39:50.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 1 -- 172.21.15.107:0/721865965 wait complete. 2024-03-25T19:39:50.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:50.912+0000 7ff6527d6640 1 librados: shutdown 2024-03-25T19:39:50.925 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:57.732 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:57.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.849+0000 7f6ea83b5640 10 monclient: get_monmap_and_config 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.849+0000 7f6ea83b5640 10 monclient: build_initial_monmap 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.849+0000 7f6ea83b5640 10 monclient: monmap: 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:39:57.850100+0000 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:39:57.850100+0000 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:39:57.849 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.850+0000 7f6ea83b5640 1 Processor -- start 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.850+0000 7f6ea83b5640 1 -- start start 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.850+0000 7f6ea83b5640 10 monclient: init 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.851+0000 7f6ea83b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.851+0000 7f6ea83b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.851+0000 7f6ea83b5640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea00a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.851+0000 7f6ea83b5640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea013f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.851+0000 7f6ea83b5640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea01430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.851+0000 7f6ea83b5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea0142520 con 0x7f6ea00a98f0 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.851+0000 7f6ea83b5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea00b28a0 con 0x7f6ea01430c0 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.851+0000 7f6ea83b5640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea00b2a30 con 0x7f6ea013f710 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.851+0000 7f6ea83b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:57.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.851+0000 7f6ea83b5640 10 monclient(hunting): authenticate will time out at 4790.192084s 2024-03-25T19:39:57.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.863+0000 7f6ea5950640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6ea013f710 0x7f6ea01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37844/0 (socket says 172.21.15.107:37844) 2024-03-25T19:39:57.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.863+0000 7f6ea5950640 1 -- 172.21.15.107:0/1782093749 learned_addr learned my addr 172.21.15.107:0/1782093749 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:57.877 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.877+0000 7f6ea514f640 1 -- 172.21.15.107:0/1782093749 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648121344 0 0) 0x7f6ea00b28a0 con 0x7f6ea01430c0 2024-03-25T19:39:57.877 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.877+0000 7f6ea514f640 10 monclient(hunting): _negotiate 2024-03-25T19:39:57.877 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.877+0000 7f6ea514f640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:57.877 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.877+0000 7f6ea514f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:57.877 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.877+0000 7f6ea514f640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:39:57.877 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.877+0000 7f6ea514f640 1 -- 172.21.15.107:0/1782093749 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e8c003520 con 0x7f6ea01430c0 2024-03-25T19:39:57.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.882+0000 7f6ea514f640 1 -- 172.21.15.107:0/1782093749 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204803070 0 0) 0x7f6ea00b2a30 con 0x7f6ea013f710 2024-03-25T19:39:57.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.882+0000 7f6ea514f640 10 monclient(hunting): _negotiate 2024-03-25T19:39:57.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.882+0000 7f6ea514f640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:57.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.882+0000 7f6ea514f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:57.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.882+0000 7f6ea514f640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:39:57.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.882+0000 7f6ea514f640 1 -- 172.21.15.107:0/1782093749 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea00b28a0 con 0x7f6ea013f710 2024-03-25T19:39:57.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 1 -- 172.21.15.107:0/1782093749 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1081999794 0 0) 0x7f6ea00b28a0 con 0x7f6ea013f710 2024-03-25T19:39:57.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 1 -- 172.21.15.107:0/1782093749 >> v1:172.21.15.195:6789/0 conn(0x7f6ea01430c0 legacy=0x7f6ea014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 1 -- 172.21.15.107:0/1782093749 >> v1:172.21.15.107:6789/0 conn(0x7f6ea00a98f0 legacy=0x7f6ea00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 10 monclient: _finish_hunting 0 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 1 monclient: found mon.noname-b 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 1 -- 172.21.15.107:0/1782093749 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea00b44c0 con 0x7f6ea013f710 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 10 monclient: _finish_auth 0 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 10 monclient: _check_auth_tickets 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 1 -- 172.21.15.107:0/1782093749 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6e90001670 con 0x7f6ea013f710 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.892+0000 7f6ea514f640 10 monclient: dump: 2024-03-25T19:39:57.893 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:57.894 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:57.894 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:57.894 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:57.894 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:57.894 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:57.894 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:57.894 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:57.894 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:57.894 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:57.894 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.893+0000 7f6ea83b5640 5 monclient: authenticate success, global_id 4409 2024-03-25T19:39:57.894 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.893+0000 7f6ea83b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:57.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.904+0000 7f6ea514f640 1 -- 172.21.15.107:0/1782093749 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e900017d0 con 0x7f6ea013f710 2024-03-25T19:39:57.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.904+0000 7f6ea514f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:57.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.904+0000 7f6ea514f640 1 -- 172.21.15.107:0/1782093749 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6e90001ba0 con 0x7f6ea013f710 2024-03-25T19:39:57.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.904+0000 7f6ea83b5640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:57.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.904+0000 7f6ea83b5640 4 set_mon_vals no callback set 2024-03-25T19:39:57.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.904+0000 7f6ea514f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:57.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.904+0000 7f6ea514f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:57.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.904+0000 7f6ea514f640 10 monclient: dump: 2024-03-25T19:39:57.904 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:57.904 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:57.904 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:57.905 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:57.905 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:57.905 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:57.905 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:57.905 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:57.905 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:57.905 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:57.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.905+0000 7f6ea83b5640 10 monclient: shutdown 2024-03-25T19:39:57.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.905+0000 7f6ea83b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:57.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.905+0000 7f6ea83b5640 1 -- 172.21.15.107:0/1782093749 >> v1:172.21.15.107:6790/0 conn(0x7f6ea013f710 legacy=0x7f6ea01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:57.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.905+0000 7f6ea83b5640 1 -- 172.21.15.107:0/1782093749 shutdown_connections 2024-03-25T19:39:57.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.905+0000 7f6ea83b5640 1 -- 172.21.15.107:0/1782093749 >> 172.21.15.107:0/1782093749 conn(0x7f6ea00b7ea0 msgr2=0x7f6ea00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:57.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.905+0000 7f6ea83b5640 1 -- 172.21.15.107:0/1782093749 shutdown_connections 2024-03-25T19:39:57.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.906+0000 7f6ea83b5640 1 -- 172.21.15.107:0/1782093749 wait complete. 2024-03-25T19:39:57.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.906+0000 7f6ea83b5640 10 monclient: build_initial_monmap 2024-03-25T19:39:57.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.906+0000 7f6ea83b5640 10 monclient: monmap: 2024-03-25T19:39:57.907 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:39:57.907 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:57.907 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:39:57.907 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:39:57.907 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:57.907 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:39:57.907 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:39:57.907 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:39:57.907 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:39:57.907 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:57.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 1 librados: starting msgr at 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 1 librados: starting objecter 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 1 Processor -- start 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 1 -- start start 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 1 librados: setting wanted keys 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 1 librados: calling monclient init 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 10 monclient: init 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea00a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea013f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea01430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea0139520 con 0x7f6ea01430c0 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea0139800 con 0x7f6ea013f710 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea0134de0 con 0x7f6ea00a98f0 2024-03-25T19:39:57.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:57.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.907+0000 7f6ea83b5640 10 monclient(hunting): authenticate will time out at 4790.248740s 2024-03-25T19:39:57.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.915+0000 7f6ea5950640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6ea013f710 0x7f6ea01d77d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:56800/0 (socket says 172.21.15.107:56800) 2024-03-25T19:39:57.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.915+0000 7f6ea5950640 1 -- 172.21.15.107:0/226908315 learned_addr learned my addr 172.21.15.107:0/226908315 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:39:57.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.918+0000 7f6e96ffd640 10 client.?.objecter ms_handle_connect 0x7f6ea013f710 2024-03-25T19:39:57.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.918+0000 7f6e96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:57.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.930+0000 7f6e96ffd640 10 client.?.objecter ms_handle_connect 0x7f6ea01430c0 2024-03-25T19:39:57.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.930+0000 7f6e96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:57.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.931+0000 7f6e96ffd640 10 client.?.objecter ms_handle_connect 0x7f6ea00a98f0 2024-03-25T19:39:57.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.931+0000 7f6e96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:57.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.931+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949274913 0 0) 0x7f6ea0139800 con 0x7f6ea013f710 2024-03-25T19:39:57.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.931+0000 7f6e96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:57.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.931+0000 7f6e96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:57.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.931+0000 7f6e96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:57.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.931+0000 7f6e96ffd640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:39:57.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.931+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e84004eb0 con 0x7f6ea013f710 2024-03-25T19:39:57.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.939+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2381831278 0 0) 0x7f6e84004eb0 con 0x7f6ea013f710 2024-03-25T19:39:57.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.939+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ea0139800 con 0x7f6ea013f710 2024-03-25T19:39:57.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.941+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062146199 0 0) 0x7f6ea0139520 con 0x7f6ea01430c0 2024-03-25T19:39:57.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.941+0000 7f6e96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:57.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.941+0000 7f6e96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:57.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.941+0000 7f6e96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:57.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.941+0000 7f6e96ffd640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:39:57.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.941+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e84004eb0 con 0x7f6ea01430c0 2024-03-25T19:39:57.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.957+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827046663 0 0) 0x7f6ea0134de0 con 0x7f6ea00a98f0 2024-03-25T19:39:57.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.957+0000 7f6e96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:57.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.957+0000 7f6e96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:57.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.957+0000 7f6e96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:57.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.957+0000 7f6e96ffd640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:39:57.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.957+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea0139520 con 0x7f6ea00a98f0 2024-03-25T19:39:57.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6e900047e0 con 0x7f6ea013f710 2024-03-25T19:39:57.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:57.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:57.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:57.960 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:57.960 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2328767625 0 0) 0x7f6ea0139800 con 0x7f6ea013f710 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 >> v1:172.21.15.107:6790/0 conn(0x7f6ea00a98f0 legacy=0x7f6ea0138470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 >> v1:172.21.15.107:6789/0 conn(0x7f6ea01430c0 legacy=0x7f6ea0133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:57.961 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 1 monclient: found mon.b 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea0139080 con 0x7f6ea013f710 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6e96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6ea83b5640 5 monclient: authenticate success, global_id 4399 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6ea83b5640 10 monclient: _renew_subs 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6ea83b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6ea83b5640 1 -- 172.21.15.107:0/226908315 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea0138b80 con 0x7f6ea013f710 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6ea83b5640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6ea83b5640 10 monclient: _renew_subs 2024-03-25T19:39:57.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6ea83b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:57.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6ea83b5640 1 -- 172.21.15.107:0/226908315 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea01d80c0 con 0x7f6ea013f710 2024-03-25T19:39:57.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.959+0000 7f6ea83b5640 1 librados: init done 2024-03-25T19:39:57.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.960+0000 7f6ea83b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:57.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.960+0000 7f6ea83b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:57.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.960+0000 7f6ea83b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:57.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.960+0000 7f6ea83b5640 1 -- 172.21.15.107:0/226908315 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea0139290 con 0x7f6ea013f710 2024-03-25T19:39:57.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.966+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e90004bf0 con 0x7f6ea013f710 2024-03-25T19:39:57.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.966+0000 7f6e96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:57.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:57.966+0000 7f6ea494e640 4 set_mon_vals no callback set 2024-03-25T19:39:58.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.237+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6e90004f00 con 0x7f6ea013f710 2024-03-25T19:39:58.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.237+0000 7f6e96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.237+0000 7f6e96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.237+0000 7f6e96ffd640 10 monclient: dump: 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:39:58.237 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:39:58.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.238+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f6e9001aed0 con 0x7f6ea013f710 2024-03-25T19:39:58.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.238+0000 7f6ea6151640 10 monclient: get_auth_request con 0x7f6e8406de80 auth_method 0 2024-03-25T19:39:58.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.238+0000 7f6e96ffd640 10 client.4399.objecter ms_handle_connect 0x7f6e8406de80 2024-03-25T19:39:58.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.240+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f6e90054ad0 con 0x7f6ea013f710 2024-03-25T19:39:58.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.240+0000 7f6e96ffd640 10 client.4399.objecter ms_dispatch 0x7f6ea0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:58.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.240+0000 7f6e96ffd640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:58.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.240+0000 7f6e96ffd640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:58.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.240+0000 7f6e96ffd640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-25T19:39:58.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.243+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e90050410 con 0x7f6ea013f710 2024-03-25T19:39:58.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.243+0000 7f6e96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:58.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.243+0000 7f6e96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:58.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.462+0000 7f6ea83b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:58.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.463+0000 7f6ea83b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:58.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.463+0000 7f6ea83b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:39:58.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.463+0000 7f6ea83b5640 1 -- 172.21.15.107:0/226908315 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ea0138d10 con 0x7f6ea013f710 2024-03-25T19:39:58.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.501+0000 7f6e96ffd640 1 -- 172.21.15.107:0/226908315 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f6e90052510 con 0x7f6ea013f710 2024-03-25T19:39:58.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.501+0000 7f6e96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:58.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.501+0000 7f6e96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:58.501 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:39:58.501 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:58.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 10 librados: watch_flush enter 2024-03-25T19:39:58.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 10 librados: watch_flush exit 2024-03-25T19:39:58.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-25T19:39:58.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 10 client.4399.objecter successfully canceled tick 2024-03-25T19:39:58.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 1 -- 172.21.15.107:0/226908315 >> v1:172.21.15.195:6800/2984408421 conn(0x7f6e8406de80 legacy=0x7f6e84070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:58.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 10 monclient: shutdown 2024-03-25T19:39:58.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:58.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 1 -- 172.21.15.107:0/226908315 >> v1:172.21.15.195:6789/0 conn(0x7f6ea013f710 legacy=0x7f6ea01d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:58.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 1 -- 172.21.15.107:0/226908315 shutdown_connections 2024-03-25T19:39:58.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 1 -- 172.21.15.107:0/226908315 >> 172.21.15.107:0/226908315 conn(0x7f6ea00b7ea0 msgr2=0x7f6ea0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:58.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 1 -- 172.21.15.107:0/226908315 shutdown_connections 2024-03-25T19:39:58.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 1 -- 172.21.15.107:0/226908315 wait complete. 2024-03-25T19:39:58.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:39:58.504+0000 7f6ea83b5640 1 librados: shutdown 2024-03-25T19:39:58.516 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:04.982 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:04:35.298 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:04.991 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:04:35.307 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:05.024 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:04:35.341 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:05.035 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:04:35.351 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:05.044 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:04:35.331 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:05.045 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:04:35.361 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:05.049 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:04:35.336 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:05.072 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:04:35.388 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:05.073 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:04:35.360 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:05.322 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:05.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.431+0000 7f6aedcdd640 10 monclient: get_monmap_and_config 2024-03-25T19:40:05.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.431+0000 7f6aedcdd640 10 monclient: build_initial_monmap 2024-03-25T19:40:05.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.431+0000 7f6aedcdd640 10 monclient: monmap: 2024-03-25T19:40:05.431 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:05.431 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:05.431 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:40:05.432593+0000 2024-03-25T19:40:05.431 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:40:05.432593+0000 2024-03-25T19:40:05.431 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:05.431 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:05.432 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:05.432 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:40:05.432 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:40:05.432 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:05.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.432+0000 7f6aedcdd640 1 Processor -- start 2024-03-25T19:40:05.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.432+0000 7f6aedcdd640 1 -- start start 2024-03-25T19:40:05.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.432+0000 7f6aedcdd640 10 monclient: init 2024-03-25T19:40:05.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.433+0000 7f6aedcdd640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:05.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.433+0000 7f6aedcdd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:05.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.433+0000 7f6aedcdd640 10 monclient(hunting): picked mon.noname-c con 0x7f6ae80a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:05.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.433+0000 7f6aedcdd640 10 monclient(hunting): picked mon.noname-b con 0x7f6ae813f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:05.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.433+0000 7f6aedcdd640 10 monclient(hunting): picked mon.noname-a con 0x7f6ae81430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:05.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.433+0000 7f6aedcdd640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae8142520 con 0x7f6ae81430c0 2024-03-25T19:40:05.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.433+0000 7f6aedcdd640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae80b28a0 con 0x7f6ae80a98f0 2024-03-25T19:40:05.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.433+0000 7f6aedcdd640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae80b2a30 con 0x7f6ae813f710 2024-03-25T19:40:05.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.433+0000 7f6aedcdd640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:05.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.433+0000 7f6aedcdd640 10 monclient(hunting): authenticate will time out at 4797.774095s 2024-03-25T19:40:05.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.447+0000 7f6ae77fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6ae80a98f0 0x7f6ae80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:58848/0 (socket says 172.21.15.107:58848) 2024-03-25T19:40:05.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.447+0000 7f6ae77fe640 1 -- 172.21.15.107:0/4211632597 learned_addr learned my addr 172.21.15.107:0/4211632597 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:05.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.464+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871642349 0 0) 0x7f6ae80b28a0 con 0x7f6ae80a98f0 2024-03-25T19:40:05.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.464+0000 7f6ae67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:05.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.464+0000 7f6ae67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:05.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.464+0000 7f6ae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:05.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.464+0000 7f6ae67fc640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:40:05.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.464+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad0003520 con 0x7f6ae80a98f0 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059889634 0 0) 0x7f6ae8142520 con 0x7f6ae81430c0 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae80b28a0 con 0x7f6ae81430c0 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881702626 0 0) 0x7f6ae80b2a30 con 0x7f6ae813f710 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:40:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.466+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae8142520 con 0x7f6ae813f710 2024-03-25T19:40:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1338161797 0 0) 0x7f6ae80b28a0 con 0x7f6ae81430c0 2024-03-25T19:40:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 >> v1:172.21.15.107:6790/0 conn(0x7f6ae813f710 legacy=0x7f6ae81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 >> v1:172.21.15.195:6789/0 conn(0x7f6ae80a98f0 legacy=0x7f6ae80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 1 monclient: found mon.noname-a 2024-03-25T19:40:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae80b44c0 con 0x7f6ae81430c0 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6ae00019f0 con 0x7f6ae81430c0 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6aedcdd640 5 monclient: authenticate success, global_id 4413 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6aedcdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6ae67fc640 10 monclient: dump: 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.478 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.479 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:05.479 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:05.479 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:05.479 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:05.479 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:05.479 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:05.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.477+0000 7f6aedcdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.483+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ae0001b50 con 0x7f6ae81430c0 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.483+0000 7f6ae67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.483+0000 7f6ae67fc640 1 -- 172.21.15.107:0/4211632597 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6ae0001f60 con 0x7f6ae81430c0 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.483+0000 7f6ae67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.483+0000 7f6ae67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.483+0000 7f6ae67fc640 10 monclient: dump: 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:05.483 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:05.484 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:05.484 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:05.484 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:05.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.483+0000 7f6aedcdd640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:05.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.483+0000 7f6aedcdd640 4 set_mon_vals no callback set 2024-03-25T19:40:05.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.484+0000 7f6aedcdd640 10 monclient: shutdown 2024-03-25T19:40:05.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.484+0000 7f6aedcdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:05.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.484+0000 7f6aedcdd640 1 -- 172.21.15.107:0/4211632597 >> v1:172.21.15.107:6789/0 conn(0x7f6ae81430c0 legacy=0x7f6ae814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:05.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.484+0000 7f6aedcdd640 1 -- 172.21.15.107:0/4211632597 shutdown_connections 2024-03-25T19:40:05.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.484+0000 7f6aedcdd640 1 -- 172.21.15.107:0/4211632597 >> 172.21.15.107:0/4211632597 conn(0x7f6ae80b7ea0 msgr2=0x7f6ae80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:05.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.484+0000 7f6aedcdd640 1 -- 172.21.15.107:0/4211632597 shutdown_connections 2024-03-25T19:40:05.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.484+0000 7f6aedcdd640 1 -- 172.21.15.107:0/4211632597 wait complete. 2024-03-25T19:40:05.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.485+0000 7f6aedcdd640 10 monclient: build_initial_monmap 2024-03-25T19:40:05.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.485+0000 7f6aedcdd640 10 monclient: monmap: 2024-03-25T19:40:05.485 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.485+0000 7f6aedcdd640 1 librados: starting msgr at 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.485+0000 7f6aedcdd640 1 librados: starting objecter 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.485+0000 7f6aedcdd640 1 Processor -- start 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.485+0000 7f6aedcdd640 1 -- start start 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.485+0000 7f6aedcdd640 1 librados: setting wanted keys 2024-03-25T19:40:05.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.485+0000 7f6aedcdd640 1 librados: calling monclient init 2024-03-25T19:40:05.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.485+0000 7f6aedcdd640 10 monclient: init 2024-03-25T19:40:05.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.486+0000 7f6aedcdd640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:05.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.486+0000 7f6aedcdd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:05.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.486+0000 7f6aedcdd640 10 monclient(hunting): picked mon.noname-a con 0x7f6ae80a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:05.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.486+0000 7f6aedcdd640 10 monclient(hunting): picked mon.noname-b con 0x7f6ae813f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:05.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.486+0000 7f6aedcdd640 10 monclient(hunting): picked mon.noname-c con 0x7f6ae81430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:05.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.486+0000 7f6aedcdd640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae81d48f0 con 0x7f6ae80a98f0 2024-03-25T19:40:05.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.486+0000 7f6aedcdd640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae81397d0 con 0x7f6ae813f710 2024-03-25T19:40:05.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.486+0000 7f6aedcdd640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae81d4b50 con 0x7f6ae81430c0 2024-03-25T19:40:05.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.486+0000 7f6aedcdd640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:05.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.486+0000 7f6aedcdd640 10 monclient(hunting): authenticate will time out at 4797.827199s 2024-03-25T19:40:05.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.500+0000 7f6ae6ffd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6ae813f710 0x7f6ae80aff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:58878/0 (socket says 172.21.15.107:58878) 2024-03-25T19:40:05.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.500+0000 7f6ae6ffd640 1 -- 172.21.15.107:0/812708650 learned_addr learned my addr 172.21.15.107:0/812708650 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:05.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.502+0000 7f6acffff640 10 client.?.objecter ms_handle_connect 0x7f6ae80a98f0 2024-03-25T19:40:05.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.502+0000 7f6acffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:05.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.505+0000 7f6acffff640 10 client.?.objecter ms_handle_connect 0x7f6ae813f710 2024-03-25T19:40:05.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.505+0000 7f6acffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:05.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.520+0000 7f6acffff640 10 client.?.objecter ms_handle_connect 0x7f6ae81430c0 2024-03-25T19:40:05.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.520+0000 7f6acffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:05.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.521+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018397191 0 0) 0x7f6ae81397d0 con 0x7f6ae813f710 2024-03-25T19:40:05.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.521+0000 7f6acffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:05.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.521+0000 7f6acffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:05.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.521+0000 7f6acffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:05.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.521+0000 7f6acffff640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:40:05.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.521+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad4004eb0 con 0x7f6ae813f710 2024-03-25T19:40:05.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.530+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887506386 0 0) 0x7f6ae81d48f0 con 0x7f6ae80a98f0 2024-03-25T19:40:05.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.530+0000 7f6acffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:05.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.530+0000 7f6acffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:05.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.530+0000 7f6acffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:05.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.530+0000 7f6acffff640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:40:05.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.530+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae81397d0 con 0x7f6ae80a98f0 2024-03-25T19:40:05.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.539+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908770763 0 0) 0x7f6ae81d4b50 con 0x7f6ae81430c0 2024-03-25T19:40:05.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.539+0000 7f6acffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:05.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.539+0000 7f6acffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:05.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.539+0000 7f6acffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:05.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.539+0000 7f6acffff640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:40:05.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.539+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae81d48f0 con 0x7f6ae81430c0 2024-03-25T19:40:05.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.554+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4088850877 0 0) 0x7f6ad4004eb0 con 0x7f6ae813f710 2024-03-25T19:40:05.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.554+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ae81d4b50 con 0x7f6ae813f710 2024-03-25T19:40:05.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.554+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6adc001a10 con 0x7f6ae813f710 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.554+0000 7f6acffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.554+0000 7f6acffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.554+0000 7f6acffff640 10 monclient(hunting): dump: 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.554+0000 7f6acffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:05.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.560+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 495497059 0 0) 0x7f6ae81d48f0 con 0x7f6ae81430c0 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.560+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ad4004eb0 con 0x7f6ae81430c0 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.560+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6ae0001e60 con 0x7f6ae81430c0 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.560+0000 7f6acffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.560+0000 7f6acffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.560+0000 7f6acffff640 10 monclient(hunting): dump: 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:05.560 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.564+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2805058494 0 0) 0x7f6ae81397d0 con 0x7f6ae80a98f0 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.564+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ae81d48f0 con 0x7f6ae80a98f0 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.564+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6ad8001950 con 0x7f6ae80a98f0 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.564+0000 7f6acffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.564+0000 7f6acffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.564+0000 7f6acffff640 10 monclient(hunting): dump: 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:05.564 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:05.565 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2737657308 0 0) 0x7f6ae81d4b50 con 0x7f6ae813f710 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 >> v1:172.21.15.107:6790/0 conn(0x7f6ae81430c0 legacy=0x7f6ae81d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 >> v1:172.21.15.107:6789/0 conn(0x7f6ae80a98f0 legacy=0x7f6ae81d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6acffff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6acffff640 1 monclient: found mon.b 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6acffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6acffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae80b16b0 con 0x7f6ae813f710 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6acffff640 10 monclient: _finish_auth 0 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6acffff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6acffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:05.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6aedcdd640 5 monclient: authenticate success, global_id 4405 2024-03-25T19:40:05.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6aedcdd640 10 monclient: _renew_subs 2024-03-25T19:40:05.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6aedcdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:05.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6aedcdd640 1 -- 172.21.15.107:0/812708650 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae80b0c10 con 0x7f6ae813f710 2024-03-25T19:40:05.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6aedcdd640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:05.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6aedcdd640 10 monclient: _renew_subs 2024-03-25T19:40:05.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6aedcdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:05.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6aedcdd640 1 -- 172.21.15.107:0/812708650 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae80b0fb0 con 0x7f6ae813f710 2024-03-25T19:40:05.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.583+0000 7f6aedcdd640 1 librados: init done 2024-03-25T19:40:05.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.585+0000 7f6aedcdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:05.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.585+0000 7f6aedcdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:05.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.585+0000 7f6aedcdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:05.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.585+0000 7f6aedcdd640 1 -- 172.21.15.107:0/812708650 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae81d4ce0 con 0x7f6ae813f710 2024-03-25T19:40:05.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.862+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6adc001c70 con 0x7f6ae813f710 2024-03-25T19:40:05.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.862+0000 7f6acffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.862+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6adc003a60 con 0x7f6ae813f710 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.862+0000 7f6acffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.862+0000 7f6acffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.862+0000 7f6acffff640 10 monclient: dump: 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:05.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.862+0000 7f6ae5ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:05.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.863+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f6adc003c80 con 0x7f6ae813f710 2024-03-25T19:40:05.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.864+0000 7f6ae77fe640 10 monclient: get_auth_request con 0x7f6ad406e0f0 auth_method 0 2024-03-25T19:40:05.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.864+0000 7f6acffff640 10 client.4405.objecter ms_handle_connect 0x7f6ad406e0f0 2024-03-25T19:40:05.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.869+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f6adc054860 con 0x7f6ae813f710 2024-03-25T19:40:05.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.869+0000 7f6acffff640 10 client.4405.objecter ms_dispatch 0x7f6ae8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:05.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.869+0000 7f6acffff640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:05.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.869+0000 7f6acffff640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:05.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.869+0000 7f6acffff640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T19:40:05.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.869+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6adc050370 con 0x7f6ae813f710 2024-03-25T19:40:05.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.869+0000 7f6acffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:05.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:05.869+0000 7f6acffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:06.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.115+0000 7f6aedcdd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:06.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.115+0000 7f6aedcdd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:06.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.115+0000 7f6aedcdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:06.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.115+0000 7f6aedcdd640 1 -- 172.21.15.107:0/812708650 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ae80627b0 con 0x7f6ae813f710 2024-03-25T19:40:06.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.160+0000 7f6acffff640 1 -- 172.21.15.107:0/812708650 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f6adc0522a0 con 0x7f6ae813f710 2024-03-25T19:40:06.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.161+0000 7f6acffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:06.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.161+0000 7f6acffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:06.160 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:40:06.161 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:06.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.163+0000 7f6aedcdd640 10 librados: watch_flush enter 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.163+0000 7f6aedcdd640 10 librados: watch_flush exit 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.163+0000 7f6aedcdd640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.163+0000 7f6aedcdd640 10 client.4405.objecter successfully canceled tick 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.163+0000 7f6aedcdd640 1 -- 172.21.15.107:0/812708650 >> v1:172.21.15.195:6800/2984408421 conn(0x7f6ad406e0f0 legacy=0x7f6ad40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.163+0000 7f6aedcdd640 10 monclient: shutdown 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.163+0000 7f6aedcdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.163+0000 7f6aedcdd640 1 -- 172.21.15.107:0/812708650 >> v1:172.21.15.195:6789/0 conn(0x7f6ae813f710 legacy=0x7f6ae80aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.164+0000 7f6aedcdd640 1 -- 172.21.15.107:0/812708650 shutdown_connections 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.164+0000 7f6aedcdd640 1 -- 172.21.15.107:0/812708650 >> 172.21.15.107:0/812708650 conn(0x7f6ae80b7ea0 msgr2=0x7f6ae8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.164+0000 7f6aedcdd640 1 -- 172.21.15.107:0/812708650 shutdown_connections 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.164+0000 7f6aedcdd640 1 -- 172.21.15.107:0/812708650 wait complete. 2024-03-25T19:40:06.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:06.164+0000 7f6aedcdd640 1 librados: shutdown 2024-03-25T19:40:06.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:12.981 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:13.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.094+0000 7ff118baf640 10 monclient: get_monmap_and_config 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.094+0000 7ff118baf640 10 monclient: build_initial_monmap 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.094+0000 7ff118baf640 10 monclient: monmap: 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:40:13.095320+0000 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:40:13.095320+0000 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:40:13.094 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.095+0000 7ff118baf640 1 Processor -- start 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.095+0000 7ff118baf640 1 -- start start 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.095+0000 7ff118baf640 10 monclient: init 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.095+0000 7ff118baf640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.095+0000 7ff118baf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.095+0000 7ff118baf640 10 monclient(hunting): picked mon.noname-a con 0x7ff1140b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.095+0000 7ff118baf640 10 monclient(hunting): picked mon.noname-b con 0x7ff1140b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.095+0000 7ff118baf640 10 monclient(hunting): picked mon.noname-c con 0x7ff11414be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.095+0000 7ff118baf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1140af940 con 0x7ff1140b4d60 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.095+0000 7ff118baf640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1140b0a70 con 0x7ff11414be90 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.096+0000 7ff118baf640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1140b0d50 con 0x7ff1140b1730 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.096+0000 7ff118baf640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:13.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.096+0000 7ff118baf640 10 monclient(hunting): authenticate will time out at 4805.436873s 2024-03-25T19:40:13.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.102+0000 7ff11259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff1140b4d60 0x7ff11413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37808/0 (socket says 172.21.15.107:37808) 2024-03-25T19:40:13.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.102+0000 7ff11259c640 1 -- 172.21.15.107:0/3379903647 learned_addr learned my addr 172.21.15.107:0/3379903647 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:13.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.108+0000 7ff111d9b640 1 -- 172.21.15.107:0/3379903647 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070328541 0 0) 0x7ff1140af940 con 0x7ff1140b4d60 2024-03-25T19:40:13.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.108+0000 7ff111d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:13.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.108+0000 7ff111d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:13.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.108+0000 7ff111d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:13.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.108+0000 7ff111d9b640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:40:13.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.108+0000 7ff111d9b640 1 -- 172.21.15.107:0/3379903647 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff100003520 con 0x7ff1140b4d60 2024-03-25T19:40:13.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 1 -- 172.21.15.107:0/3379903647 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 40207541 0 0) 0x7ff100003520 con 0x7ff1140b4d60 2024-03-25T19:40:13.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 1 -- 172.21.15.107:0/3379903647 >> v1:172.21.15.107:6790/0 conn(0x7ff1140b1730 legacy=0x7ff1140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:13.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 1 -- 172.21.15.107:0/3379903647 >> v1:172.21.15.195:6789/0 conn(0x7ff11414be90 legacy=0x7ff11414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:13.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:40:13.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 1 monclient: found mon.noname-a 2024-03-25T19:40:13.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:13.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:40:13.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 1 -- 172.21.15.107:0/3379903647 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1140b1030 con 0x7ff1140b4d60 2024-03-25T19:40:13.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 10 monclient: _finish_auth 0 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 1 -- 172.21.15.107:0/3379903647 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff104001690 con 0x7ff1140b4d60 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff111d9b640 10 monclient: dump: 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:13.118 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:13.119 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:13.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff118baf640 5 monclient: authenticate success, global_id 4419 2024-03-25T19:40:13.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.117+0000 7ff118baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:13.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.122+0000 7ff111d9b640 1 -- 172.21.15.107:0/3379903647 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1040017f0 con 0x7ff1140b4d60 2024-03-25T19:40:13.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.122+0000 7ff111d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:13.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.122+0000 7ff118baf640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:13.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.122+0000 7ff118baf640 4 set_mon_vals no callback set 2024-03-25T19:40:13.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.122+0000 7ff111d9b640 1 -- 172.21.15.107:0/3379903647 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff104001c00 con 0x7ff1140b4d60 2024-03-25T19:40:13.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.122+0000 7ff111d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:13.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.122+0000 7ff111d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:13.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.122+0000 7ff111d9b640 10 monclient: dump: 2024-03-25T19:40:13.122 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:13.123 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:13.123 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:13.123 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:13.123 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:13.123 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:13.123 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:13.123 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:13.123 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:13.123 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.123+0000 7ff118baf640 10 monclient: shutdown 2024-03-25T19:40:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.123+0000 7ff118baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.123+0000 7ff118baf640 1 -- 172.21.15.107:0/3379903647 >> v1:172.21.15.107:6789/0 conn(0x7ff1140b4d60 legacy=0x7ff11413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.123+0000 7ff118baf640 1 -- 172.21.15.107:0/3379903647 shutdown_connections 2024-03-25T19:40:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.123+0000 7ff118baf640 1 -- 172.21.15.107:0/3379903647 >> 172.21.15.107:0/3379903647 conn(0x7ff1140617a0 msgr2=0x7ff114061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.123+0000 7ff118baf640 1 -- 172.21.15.107:0/3379903647 shutdown_connections 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.123+0000 7ff118baf640 1 -- 172.21.15.107:0/3379903647 wait complete. 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.124+0000 7ff118baf640 10 monclient: build_initial_monmap 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.124+0000 7ff118baf640 10 monclient: monmap: 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:40:13.126 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.124+0000 7ff118baf640 1 librados: starting msgr at 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.124+0000 7ff118baf640 1 librados: starting objecter 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 1 Processor -- start 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 1 -- start start 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 1 librados: setting wanted keys 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 1 librados: calling monclient init 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 10 monclient: init 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 10 monclient(hunting): picked mon.noname-c con 0x7ff1140b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 10 monclient(hunting): picked mon.noname-a con 0x7ff1140b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 10 monclient(hunting): picked mon.noname-b con 0x7ff11414be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11413bde0 con 0x7ff1140b4d60 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff114140f50 con 0x7ff11414be90 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1141410e0 con 0x7ff1140b1730 2024-03-25T19:40:13.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:13.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.125+0000 7ff118baf640 10 monclient(hunting): authenticate will time out at 4805.466477s 2024-03-25T19:40:13.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.128+0000 7ff11259c640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff1140b1730 0x7ff114140840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45682/0 (socket says 172.21.15.107:45682) 2024-03-25T19:40:13.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.128+0000 7ff11259c640 1 -- 172.21.15.107:0/4011379861 learned_addr learned my addr 172.21.15.107:0/4011379861 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:13.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.130+0000 7ff10affd640 10 client.?.objecter ms_handle_connect 0x7ff1140b1730 2024-03-25T19:40:13.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.130+0000 7ff10affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:13.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.131+0000 7ff10affd640 10 client.?.objecter ms_handle_connect 0x7ff1140b4d60 2024-03-25T19:40:13.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.131+0000 7ff10affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:13.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.138+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158654211 0 0) 0x7ff11413bde0 con 0x7ff1140b4d60 2024-03-25T19:40:13.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.138+0000 7ff10affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:13.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.138+0000 7ff10affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:13.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.138+0000 7ff10affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:13.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.138+0000 7ff10affd640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:40:13.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.138+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0f4004eb0 con 0x7ff1140b4d60 2024-03-25T19:40:13.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.142+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194316276 0 0) 0x7ff1141410e0 con 0x7ff1140b1730 2024-03-25T19:40:13.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.142+0000 7ff10affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:13.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.142+0000 7ff10affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:13.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.142+0000 7ff10affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:13.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.142+0000 7ff10affd640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:40:13.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.142+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11413bde0 con 0x7ff1140b1730 2024-03-25T19:40:13.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.146+0000 7ff10affd640 10 client.?.objecter ms_handle_connect 0x7ff11414be90 2024-03-25T19:40:13.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.146+0000 7ff10affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:13.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.147+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 580870668 0 0) 0x7ff0f4004eb0 con 0x7ff1140b4d60 2024-03-25T19:40:13.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.147+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1141410e0 con 0x7ff1140b4d60 2024-03-25T19:40:13.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.156+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 960179102 0 0) 0x7ff11413bde0 con 0x7ff1140b1730 2024-03-25T19:40:13.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.156+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0f4004eb0 con 0x7ff1140b1730 2024-03-25T19:40:13.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.156+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff104001c00 con 0x7ff1140b1730 2024-03-25T19:40:13.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.156+0000 7ff10affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:13.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.156+0000 7ff10affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:13.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.156+0000 7ff10affd640 10 monclient(hunting): dump: 2024-03-25T19:40:13.156 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:13.156 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:13.156 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:13.157 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:13.157 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:13.157 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:13.157 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:13.157 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:13.157 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:13.157 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:13.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.156+0000 7ff10affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:13.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff0fc0019d0 con 0x7ff1140b4d60 2024-03-25T19:40:13.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 10 monclient(hunting): dump: 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2875236676 0 0) 0x7ff1141410e0 con 0x7ff1140b4d60 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 >> v1:172.21.15.107:6790/0 conn(0x7ff1140b1730 legacy=0x7ff114140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:13.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 >> v1:172.21.15.195:6789/0 conn(0x7ff11414be90 legacy=0x7ff114142bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 1 monclient: found mon.a 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1141d8450 con 0x7ff1140b4d60 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 10 monclient: _finish_auth 0 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff10affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff118baf640 5 monclient: authenticate success, global_id 4422 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff118baf640 10 monclient: _renew_subs 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff118baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff118baf640 1 -- 172.21.15.107:0/4011379861 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1141d86a0 con 0x7ff1140b4d60 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff118baf640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff118baf640 10 monclient: _renew_subs 2024-03-25T19:40:13.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff118baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:13.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff118baf640 1 -- 172.21.15.107:0/4011379861 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1141426d0 con 0x7ff1140b4d60 2024-03-25T19:40:13.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.157+0000 7ff118baf640 1 librados: init done 2024-03-25T19:40:13.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.158+0000 7ff118baf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:13.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.158+0000 7ff118baf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:13.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.158+0000 7ff118baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:13.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.158+0000 7ff118baf640 1 -- 172.21.15.107:0/4011379861 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1141401f0 con 0x7ff1140b4d60 2024-03-25T19:40:13.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.464+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0fc0035f0 con 0x7ff1140b4d60 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.464+0000 7ff10affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.464+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff0fc0039c0 con 0x7ff1140b4d60 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.464+0000 7ff10affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.464+0000 7ff10affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.464+0000 7ff10affd640 10 monclient: dump: 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:13.464 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:13.465 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:13.465 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:13.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.464+0000 7ff11159a640 4 set_mon_vals no callback set 2024-03-25T19:40:13.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.465+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7ff0fc003be0 con 0x7ff1140b4d60 2024-03-25T19:40:13.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.465+0000 7ff11259c640 10 monclient: get_auth_request con 0x7ff0f406def0 auth_method 0 2024-03-25T19:40:13.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.466+0000 7ff10affd640 10 client.4422.objecter ms_handle_connect 0x7ff0f406def0 2024-03-25T19:40:13.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.475+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7ff0fc053590 con 0x7ff1140b4d60 2024-03-25T19:40:13.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.475+0000 7ff10affd640 10 client.4422.objecter ms_dispatch 0x7ff114000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:13.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.475+0000 7ff10affd640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:13.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.475+0000 7ff10affd640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:13.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.475+0000 7ff10affd640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T19:40:13.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.475+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0fc04f0a0 con 0x7ff1140b4d60 2024-03-25T19:40:13.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.475+0000 7ff10affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:13.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.475+0000 7ff10affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:13.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.700+0000 7ff118baf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:13.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.700+0000 7ff118baf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:13.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.700+0000 7ff118baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:13.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.700+0000 7ff118baf640 1 -- 172.21.15.107:0/4011379861 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1141d7a60 con 0x7ff1140b4d60 2024-03-25T19:40:13.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.724+0000 7ff10affd640 1 -- 172.21.15.107:0/4011379861 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7ff1141d7a60 con 0x7ff1140b4d60 2024-03-25T19:40:13.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.724+0000 7ff10affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:13.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.724+0000 7ff10affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:13.724 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:40:13.724 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:13.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 10 librados: watch_flush enter 2024-03-25T19:40:13.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 10 librados: watch_flush exit 2024-03-25T19:40:13.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-25T19:40:13.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 10 client.4422.objecter successfully canceled tick 2024-03-25T19:40:13.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 1 -- 172.21.15.107:0/4011379861 >> v1:172.21.15.195:6800/2984408421 conn(0x7ff0f406def0 legacy=0x7ff0f40703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:13.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 10 monclient: shutdown 2024-03-25T19:40:13.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:13.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 1 -- 172.21.15.107:0/4011379861 >> v1:172.21.15.107:6789/0 conn(0x7ff1140b4d60 legacy=0x7ff1141d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:13.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 1 -- 172.21.15.107:0/4011379861 shutdown_connections 2024-03-25T19:40:13.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 1 -- 172.21.15.107:0/4011379861 >> 172.21.15.107:0/4011379861 conn(0x7ff1140617a0 msgr2=0x7ff1140b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:13.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 1 -- 172.21.15.107:0/4011379861 shutdown_connections 2024-03-25T19:40:13.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 1 -- 172.21.15.107:0/4011379861 wait complete. 2024-03-25T19:40:13.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:13.726+0000 7ff118baf640 1 librados: shutdown 2024-03-25T19:40:13.737 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:20.544 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:20.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.658+0000 7f68b93ed640 10 monclient: get_monmap_and_config 2024-03-25T19:40:20.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.658+0000 7f68b93ed640 10 monclient: build_initial_monmap 2024-03-25T19:40:20.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.658+0000 7f68b93ed640 10 monclient: monmap: 2024-03-25T19:40:20.658 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:20.658 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:20.658 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:40:20.659590+0000 2024-03-25T19:40:20.658 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:40:20.659590+0000 2024-03-25T19:40:20.658 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:20.658 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:20.658 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:20.659 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:40:20.659 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:40:20.659 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:20.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.659+0000 7f68b93ed640 1 Processor -- start 2024-03-25T19:40:20.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.659+0000 7f68b93ed640 1 -- start start 2024-03-25T19:40:20.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.659+0000 7f68b93ed640 10 monclient: init 2024-03-25T19:40:20.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.660+0000 7f68b93ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:20.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.660+0000 7f68b93ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:20.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.660+0000 7f68b93ed640 10 monclient(hunting): picked mon.noname-a con 0x7f68b40a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:20.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.660+0000 7f68b93ed640 10 monclient(hunting): picked mon.noname-b con 0x7f68b413f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:20.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.660+0000 7f68b93ed640 10 monclient(hunting): picked mon.noname-c con 0x7f68b41430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:20.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.660+0000 7f68b93ed640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b4142520 con 0x7f68b40a98f0 2024-03-25T19:40:20.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.660+0000 7f68b93ed640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b40b28a0 con 0x7f68b41430c0 2024-03-25T19:40:20.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.660+0000 7f68b93ed640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b40b2a30 con 0x7f68b413f710 2024-03-25T19:40:20.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.660+0000 7f68b93ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:20.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.660+0000 7f68b93ed640 10 monclient(hunting): authenticate will time out at 4813.001034s 2024-03-25T19:40:20.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.664+0000 7f68b2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f68b40a98f0 0x7f68b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45102/0 (socket says 172.21.15.107:45102) 2024-03-25T19:40:20.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.664+0000 7f68b2ffd640 1 -- 172.21.15.107:0/1337655678 learned_addr learned my addr 172.21.15.107:0/1337655678 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:20.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.670+0000 7f68b1ffb640 1 -- 172.21.15.107:0/1337655678 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891721220 0 0) 0x7f68b4142520 con 0x7f68b40a98f0 2024-03-25T19:40:20.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.671+0000 7f68b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:20.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.671+0000 7f68b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:20.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.671+0000 7f68b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:20.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.671+0000 7f68b1ffb640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:40:20.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.671+0000 7f68b1ffb640 1 -- 172.21.15.107:0/1337655678 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f689c003520 con 0x7f68b40a98f0 2024-03-25T19:40:20.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 1 -- 172.21.15.107:0/1337655678 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362169338 0 0) 0x7f689c003520 con 0x7f68b40a98f0 2024-03-25T19:40:20.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 1 -- 172.21.15.107:0/1337655678 >> v1:172.21.15.107:6790/0 conn(0x7f68b413f710 legacy=0x7f68b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:40:20.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 1 -- 172.21.15.107:0/1337655678 >> v1:172.21.15.195:6789/0 conn(0x7f68b41430c0 legacy=0x7f68b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:20.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:20.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 1 monclient: found mon.noname-a 2024-03-25T19:40:20.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:20.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:40:20.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 1 -- 172.21.15.107:0/1337655678 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b40b44c0 con 0x7f68b40a98f0 2024-03-25T19:40:20.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:20.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:20.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 1 -- 172.21.15.107:0/1337655678 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f68a4001690 con 0x7f68b40a98f0 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b93ed640 5 monclient: authenticate success, global_id 4425 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b93ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b1ffb640 10 monclient: dump: 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:20.680 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.679+0000 7f68b93ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.688+0000 7f68b1ffb640 1 -- 172.21.15.107:0/1337655678 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68a4001860 con 0x7f68b40a98f0 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.688+0000 7f68b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.688+0000 7f68b1ffb640 1 -- 172.21.15.107:0/1337655678 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f68a4001cb0 con 0x7f68b40a98f0 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.688+0000 7f68b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.688+0000 7f68b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.688+0000 7f68b1ffb640 10 monclient: dump: 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:20.688 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:20.689 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:20.689 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:20.689 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:20.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.688+0000 7f68b93ed640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:20.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.688+0000 7f68b93ed640 4 set_mon_vals no callback set 2024-03-25T19:40:20.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.689+0000 7f68b93ed640 10 monclient: shutdown 2024-03-25T19:40:20.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.689+0000 7f68b93ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:20.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.689+0000 7f68b93ed640 1 -- 172.21.15.107:0/1337655678 >> v1:172.21.15.107:6789/0 conn(0x7f68b40a98f0 legacy=0x7f68b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.689+0000 7f68b93ed640 1 -- 172.21.15.107:0/1337655678 shutdown_connections 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.689+0000 7f68b93ed640 1 -- 172.21.15.107:0/1337655678 >> 172.21.15.107:0/1337655678 conn(0x7f68b40b7ea0 msgr2=0x7f68b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.689+0000 7f68b93ed640 1 -- 172.21.15.107:0/1337655678 shutdown_connections 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.689+0000 7f68b93ed640 1 -- 172.21.15.107:0/1337655678 wait complete. 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.690+0000 7f68b93ed640 10 monclient: build_initial_monmap 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.690+0000 7f68b93ed640 10 monclient: monmap: 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:40:20.692 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.690+0000 7f68b93ed640 1 librados: starting msgr at 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.690+0000 7f68b93ed640 1 librados: starting objecter 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.690+0000 7f68b93ed640 1 Processor -- start 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.691+0000 7f68b93ed640 1 -- start start 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.691+0000 7f68b93ed640 1 librados: setting wanted keys 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.691+0000 7f68b93ed640 1 librados: calling monclient init 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.691+0000 7f68b93ed640 10 monclient: init 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.691+0000 7f68b93ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.691+0000 7f68b93ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.691+0000 7f68b93ed640 10 monclient(hunting): picked mon.noname-c con 0x7f68b40a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.691+0000 7f68b93ed640 10 monclient(hunting): picked mon.noname-b con 0x7f68b413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.692+0000 7f68b93ed640 10 monclient(hunting): picked mon.noname-a con 0x7f68b41430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.692+0000 7f68b93ed640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b4136f30 con 0x7f68b41430c0 2024-03-25T19:40:20.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.692+0000 7f68b93ed640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b4135570 con 0x7f68b413f710 2024-03-25T19:40:20.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.692+0000 7f68b93ed640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b4135070 con 0x7f68b40a98f0 2024-03-25T19:40:20.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.692+0000 7f68b93ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:20.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.692+0000 7f68b93ed640 10 monclient(hunting): authenticate will time out at 4813.032900s 2024-03-25T19:40:20.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.699+0000 7f68b27fc640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f68b413f710 0x7f68b41dade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:47204/0 (socket says 172.21.15.107:47204) 2024-03-25T19:40:20.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.699+0000 7f68b27fc640 1 -- 172.21.15.107:0/2110398996 learned_addr learned my addr 172.21.15.107:0/2110398996 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:20.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.701+0000 7f689b7fe640 10 client.?.objecter ms_handle_connect 0x7f68b413f710 2024-03-25T19:40:20.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.701+0000 7f689b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:20.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.703+0000 7f689b7fe640 10 client.?.objecter ms_handle_connect 0x7f68b41430c0 2024-03-25T19:40:20.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.703+0000 7f689b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:20.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.707+0000 7f689b7fe640 10 client.?.objecter ms_handle_connect 0x7f68b40a98f0 2024-03-25T19:40:20.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.707+0000 7f689b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:20.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.709+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992077131 0 0) 0x7f68b4136f30 con 0x7f68b41430c0 2024-03-25T19:40:20.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.709+0000 7f689b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:20.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.709+0000 7f689b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:20.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.709+0000 7f689b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:20.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.709+0000 7f689b7fe640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:40:20.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.709+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a0004eb0 con 0x7f68b41430c0 2024-03-25T19:40:20.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.718+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2968045954 0 0) 0x7f68a0004eb0 con 0x7f68b41430c0 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.718+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68b4136f30 con 0x7f68b41430c0 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.718+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f68ac001b90 con 0x7f68b41430c0 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.718+0000 7f689b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.718+0000 7f689b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.718+0000 7f689b7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:20.718 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.718+0000 7f689b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.719+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811358399 0 0) 0x7f68b4135570 con 0x7f68b413f710 2024-03-25T19:40:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.719+0000 7f689b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.719+0000 7f689b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.719+0000 7f689b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.719+0000 7f689b7fe640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:40:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.719+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a0004eb0 con 0x7f68b413f710 2024-03-25T19:40:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143215063 0 0) 0x7f68b4135070 con 0x7f68b40a98f0 2024-03-25T19:40:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:40:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b4135570 con 0x7f68b40a98f0 2024-03-25T19:40:20.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3453868150 0 0) 0x7f68b4136f30 con 0x7f68b41430c0 2024-03-25T19:40:20.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 >> v1:172.21.15.107:6790/0 conn(0x7f68b40a98f0 legacy=0x7f68b41d4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:20.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 >> v1:172.21.15.195:6789/0 conn(0x7f68b413f710 legacy=0x7f68b41dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:20.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 1 monclient: found mon.a 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b41d41e0 con 0x7f68b41430c0 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f689b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f68b93ed640 5 monclient: authenticate success, global_id 4428 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f68b93ed640 10 monclient: _renew_subs 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f68b93ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.723+0000 7f68b93ed640 1 -- 172.21.15.107:0/2110398996 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b4135cf0 con 0x7f68b41430c0 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.724+0000 7f68b93ed640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.724+0000 7f68b93ed640 10 monclient: _renew_subs 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.724+0000 7f68b93ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.724+0000 7f68b93ed640 1 -- 172.21.15.107:0/2110398996 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b413ca50 con 0x7f68b41430c0 2024-03-25T19:40:20.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.724+0000 7f68b93ed640 1 librados: init done 2024-03-25T19:40:20.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.725+0000 7f68b93ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:20.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.725+0000 7f68b93ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:20.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.725+0000 7f68b93ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:20.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:20.725+0000 7f68b93ed640 1 -- 172.21.15.107:0/2110398996 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b41d43d0 con 0x7f68b41430c0 2024-03-25T19:40:21.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.024+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68ac0020c0 con 0x7f68b41430c0 2024-03-25T19:40:21.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.024+0000 7f689b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:21.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.024+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f68ac003b30 con 0x7f68b41430c0 2024-03-25T19:40:21.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.024+0000 7f689b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:21.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.024+0000 7f689b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:21.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.024+0000 7f689b7fe640 10 monclient: dump: 2024-03-25T19:40:21.122 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:21.123 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:21.123 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:21.123 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:21.123 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:21.123 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:21.123 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:21.123 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:21.123 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:21.123 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:21.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.024+0000 7f68b17fa640 4 set_mon_vals no callback set 2024-03-25T19:40:21.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.024+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f68ac003d50 con 0x7f68b41430c0 2024-03-25T19:40:21.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.025+0000 7f68b2ffd640 10 monclient: get_auth_request con 0x7f68a006de10 auth_method 0 2024-03-25T19:40:21.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.025+0000 7f689b7fe640 10 client.4428.objecter ms_handle_connect 0x7f68a006de10 2024-03-25T19:40:21.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.029+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f68ac0538c0 con 0x7f68b41430c0 2024-03-25T19:40:21.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.029+0000 7f689b7fe640 10 client.4428.objecter ms_dispatch 0x7f68b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:21.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.029+0000 7f689b7fe640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:21.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.029+0000 7f689b7fe640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:21.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.029+0000 7f689b7fe640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-25T19:40:21.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.029+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f68ac04f200 con 0x7f68b41430c0 2024-03-25T19:40:21.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.029+0000 7f689b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:21.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.029+0000 7f689b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:21.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.287+0000 7f68b93ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:21.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.287+0000 7f68b93ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:21.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.287+0000 7f68b93ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:21.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.287+0000 7f68b93ed640 1 -- 172.21.15.107:0/2110398996 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68b40b3ef0 con 0x7f68b41430c0 2024-03-25T19:40:21.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.360+0000 7f689b7fe640 1 -- 172.21.15.107:0/2110398996 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f68ac051300 con 0x7f68b41430c0 2024-03-25T19:40:21.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.360+0000 7f689b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:21.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.360+0000 7f689b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:21.360 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:40:21.360 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.362+0000 7f68b93ed640 10 librados: watch_flush enter 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 10 librados: watch_flush exit 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 10 client.4428.objecter successfully canceled tick 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 1 -- 172.21.15.107:0/2110398996 >> v1:172.21.15.195:6800/2984408421 conn(0x7f68a006de10 legacy=0x7f68a00702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 10 monclient: shutdown 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 1 -- 172.21.15.107:0/2110398996 >> v1:172.21.15.107:6789/0 conn(0x7f68b41430c0 legacy=0x7f68b41db4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 1 -- 172.21.15.107:0/2110398996 shutdown_connections 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 1 -- 172.21.15.107:0/2110398996 >> 172.21.15.107:0/2110398996 conn(0x7f68b40b7ea0 msgr2=0x7f68b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 1 -- 172.21.15.107:0/2110398996 shutdown_connections 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 1 -- 172.21.15.107:0/2110398996 wait complete. 2024-03-25T19:40:21.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:21.363+0000 7f68b93ed640 1 librados: shutdown 2024-03-25T19:40:21.374 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:25.041 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:04:55.357 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:25.096 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:04:55.413 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:25.122 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:04:55.439 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:25.146 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:04:55.433 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:25.150 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:04:55.437 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:25.151 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:04:55.468 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:25.156 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:04:55.473 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:25.184 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:04:55.471 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:25.197 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:04:55.513 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:28.180 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:28.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.281+0000 7fb8ce1a8640 10 monclient: get_monmap_and_config 2024-03-25T19:40:28.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.281+0000 7fb8ce1a8640 10 monclient: build_initial_monmap 2024-03-25T19:40:28.281 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.281+0000 7fb8ce1a8640 10 monclient: monmap: 2024-03-25T19:40:28.281 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:28.281 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:28.281 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:40:28.282227+0000 2024-03-25T19:40:28.281 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:40:28.282227+0000 2024-03-25T19:40:28.281 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:28.281 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:28.281 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:28.281 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:40:28.281 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:40:28.281 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:28.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.282+0000 7fb8ce1a8640 1 Processor -- start 2024-03-25T19:40:28.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.282+0000 7fb8ce1a8640 1 -- start start 2024-03-25T19:40:28.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.282+0000 7fb8ce1a8640 10 monclient: init 2024-03-25T19:40:28.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.283+0000 7fb8ce1a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:28.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.283+0000 7fb8ce1a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:28.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.283+0000 7fb8ce1a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb8c80b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:28.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.283+0000 7fb8ce1a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb8c80b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:28.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.283+0000 7fb8ce1a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb8c814be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:28.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.283+0000 7fb8ce1a8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c80af940 con 0x7fb8c80b4d60 2024-03-25T19:40:28.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.283+0000 7fb8ce1a8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c80b0a70 con 0x7fb8c814be90 2024-03-25T19:40:28.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.283+0000 7fb8ce1a8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c80b0d50 con 0x7fb8c80b1730 2024-03-25T19:40:28.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.283+0000 7fb8ce1a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:28.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.283+0000 7fb8ce1a8640 10 monclient(hunting): authenticate will time out at 4820.624412s 2024-03-25T19:40:28.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.292+0000 7fb8c7fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb8c814be90 0x7fb8c814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:52032/0 (socket says 172.21.15.107:52032) 2024-03-25T19:40:28.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.292+0000 7fb8c7fff640 1 -- 172.21.15.107:0/506120390 learned_addr learned my addr 172.21.15.107:0/506120390 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:28.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.308+0000 7fb8c67fc640 1 -- 172.21.15.107:0/506120390 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675238403 0 0) 0x7fb8c80b0a70 con 0x7fb8c814be90 2024-03-25T19:40:28.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.308+0000 7fb8c67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:28.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.308+0000 7fb8c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:28.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.308+0000 7fb8c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:28.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.308+0000 7fb8c67fc640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:40:28.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.308+0000 7fb8c67fc640 1 -- 172.21.15.107:0/506120390 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b0003520 con 0x7fb8c814be90 2024-03-25T19:40:28.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.309+0000 7fb8c67fc640 1 -- 172.21.15.107:0/506120390 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683316822 0 0) 0x7fb8c80b0d50 con 0x7fb8c80b1730 2024-03-25T19:40:28.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.309+0000 7fb8c67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:28.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.309+0000 7fb8c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:28.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.309+0000 7fb8c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:28.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.309+0000 7fb8c67fc640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:40:28.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.309+0000 7fb8c67fc640 1 -- 172.21.15.107:0/506120390 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c80b0a70 con 0x7fb8c80b1730 2024-03-25T19:40:28.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 1 -- 172.21.15.107:0/506120390 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2932607186 0 0) 0x7fb8b0003520 con 0x7fb8c814be90 2024-03-25T19:40:28.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 1 -- 172.21.15.107:0/506120390 >> v1:172.21.15.107:6790/0 conn(0x7fb8c80b1730 legacy=0x7fb8c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 1 -- 172.21.15.107:0/506120390 >> v1:172.21.15.107:6789/0 conn(0x7fb8c80b4d60 legacy=0x7fb8c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 1 monclient: found mon.noname-c 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 1 -- 172.21.15.107:0/506120390 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c80b1030 con 0x7fb8c814be90 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 1 -- 172.21.15.107:0/506120390 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb8c0001a10 con 0x7fb8c814be90 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8ce1a8640 5 monclient: authenticate success, global_id 4420 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8ce1a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:28.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8c67fc640 10 monclient: dump: 2024-03-25T19:40:28.321 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:28.321 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:28.321 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:28.321 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:28.321 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:28.321 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:28.321 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:28.321 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:28.321 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:28.321 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:28.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.319+0000 7fb8ce1a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:28.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.337+0000 7fb8c67fc640 1 -- 172.21.15.107:0/506120390 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8c0001b70 con 0x7fb8c814be90 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.337+0000 7fb8c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.337+0000 7fb8c67fc640 1 -- 172.21.15.107:0/506120390 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb8c0001f80 con 0x7fb8c814be90 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.337+0000 7fb8ce1a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.337+0000 7fb8ce1a8640 4 set_mon_vals no callback set 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.337+0000 7fb8c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.337+0000 7fb8c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.337+0000 7fb8c67fc640 10 monclient: dump: 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:28.337 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:28.338 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:28.338 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:28.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.338+0000 7fb8ce1a8640 10 monclient: shutdown 2024-03-25T19:40:28.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.338+0000 7fb8ce1a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:28.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.338+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/506120390 >> v1:172.21.15.195:6789/0 conn(0x7fb8c814be90 legacy=0x7fb8c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:28.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.338+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/506120390 shutdown_connections 2024-03-25T19:40:28.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.338+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/506120390 >> 172.21.15.107:0/506120390 conn(0x7fb8c80617a0 msgr2=0x7fb8c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:28.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.338+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/506120390 shutdown_connections 2024-03-25T19:40:28.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.338+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/506120390 wait complete. 2024-03-25T19:40:28.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.339+0000 7fb8ce1a8640 10 monclient: build_initial_monmap 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.339+0000 7fb8ce1a8640 10 monclient: monmap: 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.339+0000 7fb8ce1a8640 1 librados: starting msgr at 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.339+0000 7fb8ce1a8640 1 librados: starting objecter 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.339+0000 7fb8ce1a8640 1 Processor -- start 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.340+0000 7fb8ce1a8640 1 -- start start 2024-03-25T19:40:28.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.340+0000 7fb8ce1a8640 1 librados: setting wanted keys 2024-03-25T19:40:28.341 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.340+0000 7fb8ce1a8640 1 librados: calling monclient init 2024-03-25T19:40:28.341 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.340+0000 7fb8ce1a8640 10 monclient: init 2024-03-25T19:40:28.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.340+0000 7fb8ce1a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:28.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.340+0000 7fb8ce1a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:28.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.340+0000 7fb8ce1a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb8c80b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:28.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.341+0000 7fb8ce1a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb8c80b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:28.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.341+0000 7fb8ce1a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb8c814be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:28.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.341+0000 7fb8ce1a8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c8145290 con 0x7fb8c80b4d60 2024-03-25T19:40:28.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.341+0000 7fb8ce1a8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c8145500 con 0x7fb8c814be90 2024-03-25T19:40:28.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.341+0000 7fb8ce1a8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c8140520 con 0x7fb8c80b1730 2024-03-25T19:40:28.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.341+0000 7fb8ce1a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:28.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.341+0000 7fb8ce1a8640 10 monclient(hunting): authenticate will time out at 4820.681895s 2024-03-25T19:40:28.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.353+0000 7fb8c6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb8c80b4d60 0x7fb8c81409b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49888/0 (socket says 172.21.15.107:49888) 2024-03-25T19:40:28.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.353+0000 7fb8c6ffd640 1 -- 172.21.15.107:0/3912272706 learned_addr learned my addr 172.21.15.107:0/3912272706 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:28.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.354+0000 7fb8affff640 10 client.?.objecter ms_handle_connect 0x7fb8c80b4d60 2024-03-25T19:40:28.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.354+0000 7fb8affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:28.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.357+0000 7fb8affff640 10 client.?.objecter ms_handle_connect 0x7fb8c80b1730 2024-03-25T19:40:28.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.357+0000 7fb8affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:28.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.361+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250936828 0 0) 0x7fb8c8145290 con 0x7fb8c80b4d60 2024-03-25T19:40:28.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.361+0000 7fb8affff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:28.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.361+0000 7fb8affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:28.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.361+0000 7fb8affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:28.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.361+0000 7fb8affff640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:40:28.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.361+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b4004eb0 con 0x7fb8c80b4d60 2024-03-25T19:40:28.362 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.362+0000 7fb8affff640 10 client.?.objecter ms_handle_connect 0x7fb8c814be90 2024-03-25T19:40:28.362 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.362+0000 7fb8affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.369+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3456184318 0 0) 0x7fb8b4004eb0 con 0x7fb8c80b4d60 2024-03-25T19:40:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.369+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8c8145290 con 0x7fb8c80b4d60 2024-03-25T19:40:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.369+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb8bc001a10 con 0x7fb8c80b4d60 2024-03-25T19:40:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.369+0000 7fb8affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.369+0000 7fb8affff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.369+0000 7fb8affff640 10 monclient(hunting): dump: 2024-03-25T19:40:28.369 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:28.369 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:28.369 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:28.369 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:28.370 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:28.370 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:28.370 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:28.370 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:28.370 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:28.370 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:28.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.369+0000 7fb8affff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:28.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.380+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985994066 0 0) 0x7fb8c8145500 con 0x7fb8c814be90 2024-03-25T19:40:28.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.380+0000 7fb8affff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:28.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.380+0000 7fb8affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:28.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.380+0000 7fb8affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:28.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.380+0000 7fb8affff640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:40:28.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.380+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b4004eb0 con 0x7fb8c814be90 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2078153320 0 0) 0x7fb8c8145290 con 0x7fb8c80b4d60 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 >> v1:172.21.15.107:6790/0 conn(0x7fb8c80b1730 legacy=0x7fb8c8143ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 >> v1:172.21.15.195:6789/0 conn(0x7fb8c814be90 legacy=0x7fb8c81e2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8affff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8affff640 1 monclient: found mon.a 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8affff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c813ee90 con 0x7fb8c80b4d60 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8affff640 10 monclient: _finish_auth 0 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8affff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8ce1a8640 5 monclient: authenticate success, global_id 4434 2024-03-25T19:40:28.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8ce1a8640 10 monclient: _renew_subs 2024-03-25T19:40:28.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8ce1a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:28.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/3912272706 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c81457e0 con 0x7fb8c80b4d60 2024-03-25T19:40:28.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8ce1a8640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:28.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8ce1a8640 10 monclient: _renew_subs 2024-03-25T19:40:28.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8ce1a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:28.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/3912272706 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c8137270 con 0x7fb8c80b4d60 2024-03-25T19:40:28.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.383+0000 7fb8ce1a8640 1 librados: init done 2024-03-25T19:40:28.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.384+0000 7fb8ce1a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:28.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.384+0000 7fb8ce1a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:28.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.384+0000 7fb8ce1a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:28.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.384+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/3912272706 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c80627b0 con 0x7fb8c80b4d60 2024-03-25T19:40:28.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.695+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8bc001c90 con 0x7fb8c80b4d60 2024-03-25T19:40:28.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.695+0000 7fb8affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:28.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.695+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb8bc003a60 con 0x7fb8c80b4d60 2024-03-25T19:40:28.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.695+0000 7fb8affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:28.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.695+0000 7fb8affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:28.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.695+0000 7fb8affff640 10 monclient: dump: 2024-03-25T19:40:28.695 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:28.695 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:28.695 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:28.696 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:28.696 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:28.696 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:28.696 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:28.696 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:28.696 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:28.696 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:28.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.695+0000 7fb8c5ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:28.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.696+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fb8bc003c80 con 0x7fb8c80b4d60 2024-03-25T19:40:28.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.697+0000 7fb8c77fe640 10 monclient: get_auth_request con 0x7fb8b406dd00 auth_method 0 2024-03-25T19:40:28.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.697+0000 7fb8affff640 10 client.4434.objecter ms_handle_connect 0x7fb8b406dd00 2024-03-25T19:40:28.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.702+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fb8bc0548c0 con 0x7fb8c80b4d60 2024-03-25T19:40:28.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.702+0000 7fb8affff640 10 client.4434.objecter ms_dispatch 0x7fb8c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:28.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.702+0000 7fb8affff640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:28.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.702+0000 7fb8affff640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:28.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.702+0000 7fb8affff640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-25T19:40:28.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.702+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8bc0503d0 con 0x7fb8c80b4d60 2024-03-25T19:40:28.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.702+0000 7fb8affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:28.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.702+0000 7fb8affff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:28.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.951+0000 7fb8ce1a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:28.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.951+0000 7fb8ce1a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:28.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.951+0000 7fb8ce1a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:40:28.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.951+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/3912272706 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8c81438d0 con 0x7fb8c80b4d60 2024-03-25T19:40:28.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.982+0000 7fb8affff640 1 -- 172.21.15.107:0/3912272706 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fb8bc052300 con 0x7fb8c80b4d60 2024-03-25T19:40:28.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.982+0000 7fb8affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:28.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.982+0000 7fb8affff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:28.982 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:40:28.982 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:28.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.984+0000 7fb8ce1a8640 10 librados: watch_flush enter 2024-03-25T19:40:28.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 10 librados: watch_flush exit 2024-03-25T19:40:28.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-25T19:40:28.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 10 client.4434.objecter successfully canceled tick 2024-03-25T19:40:28.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/3912272706 >> v1:172.21.15.195:6800/2984408421 conn(0x7fb8b406dd00 legacy=0x7fb8b40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:28.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 10 monclient: shutdown 2024-03-25T19:40:28.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:28.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/3912272706 >> v1:172.21.15.107:6789/0 conn(0x7fb8c80b4d60 legacy=0x7fb8c81409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:28.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/3912272706 shutdown_connections 2024-03-25T19:40:28.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/3912272706 >> 172.21.15.107:0/3912272706 conn(0x7fb8c80617a0 msgr2=0x7fb8c813c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:28.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/3912272706 shutdown_connections 2024-03-25T19:40:28.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 1 -- 172.21.15.107:0/3912272706 wait complete. 2024-03-25T19:40:28.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:28.985+0000 7fb8ce1a8640 1 librados: shutdown 2024-03-25T19:40:28.996 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:35.803 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:35.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.917+0000 7f475b405640 10 monclient: get_monmap_and_config 2024-03-25T19:40:35.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.917+0000 7f475b405640 10 monclient: build_initial_monmap 2024-03-25T19:40:35.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.917+0000 7f475b405640 10 monclient: monmap: 2024-03-25T19:40:35.917 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:35.917 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:35.918 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:40:35.918680+0000 2024-03-25T19:40:35.918 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:40:35.918680+0000 2024-03-25T19:40:35.918 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:35.918 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:35.918 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:35.918 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:40:35.918 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:40:35.918 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:35.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.918+0000 7f475b405640 1 Processor -- start 2024-03-25T19:40:35.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.918+0000 7f475b405640 1 -- start start 2024-03-25T19:40:35.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.918+0000 7f475b405640 10 monclient: init 2024-03-25T19:40:35.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.919+0000 7f475b405640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:35.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.919+0000 7f475b405640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:35.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.919+0000 7f475b405640 10 monclient(hunting): picked mon.noname-a con 0x7f47540a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:35.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.919+0000 7f475b405640 10 monclient(hunting): picked mon.noname-c con 0x7f475413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:35.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.919+0000 7f475b405640 10 monclient(hunting): picked mon.noname-b con 0x7f47541430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:35.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.919+0000 7f475b405640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4754142520 con 0x7f47540a98f0 2024-03-25T19:40:35.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.919+0000 7f475b405640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47540b28a0 con 0x7f475413f710 2024-03-25T19:40:35.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.919+0000 7f475b405640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47540b2a30 con 0x7f47541430c0 2024-03-25T19:40:35.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.919+0000 7f475b405640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:35.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.919+0000 7f475b405640 10 monclient(hunting): authenticate will time out at 4828.260136s 2024-03-25T19:40:35.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.931+0000 7f47589a0640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f475413f710 0x7f47541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:40934/0 (socket says 172.21.15.107:40934) 2024-03-25T19:40:35.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.931+0000 7f47589a0640 1 -- 172.21.15.107:0/4238416656 learned_addr learned my addr 172.21.15.107:0/4238416656 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:35.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.955+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318176307 0 0) 0x7f47540b28a0 con 0x7f475413f710 2024-03-25T19:40:35.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.955+0000 7f474bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:35.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.955+0000 7f474bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:35.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.955+0000 7f474bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:35.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.955+0000 7f474bfff640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:40:35.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.955+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4740003520 con 0x7f475413f710 2024-03-25T19:40:35.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.957+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189017754 0 0) 0x7f4754142520 con 0x7f47540a98f0 2024-03-25T19:40:35.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.957+0000 7f474bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:35.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.957+0000 7f474bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:35.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.957+0000 7f474bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:35.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.957+0000 7f474bfff640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:40:35.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.957+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47540b28a0 con 0x7f47540a98f0 2024-03-25T19:40:35.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.965+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257273026 0 0) 0x7f47540b2a30 con 0x7f47541430c0 2024-03-25T19:40:35.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.965+0000 7f474bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:35.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.965+0000 7f474bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:35.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.965+0000 7f474bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:35.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.965+0000 7f474bfff640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:40:35.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.965+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4754142520 con 0x7f47541430c0 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3501924531 0 0) 0x7f4740003520 con 0x7f475413f710 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 >> v1:172.21.15.107:6790/0 conn(0x7f47541430c0 legacy=0x7f475414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 >> v1:172.21.15.107:6789/0 conn(0x7f47540a98f0 legacy=0x7f47540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 1 monclient: found mon.noname-c 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47540b44c0 con 0x7f475413f710 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 10 monclient: _finish_auth 0 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4744001690 con 0x7f475413f710 2024-03-25T19:40:35.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.966+0000 7f474bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.967+0000 7f474bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.967+0000 7f474bfff640 10 monclient: dump: 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.967+0000 7f475b405640 5 monclient: authenticate success, global_id 4426 2024-03-25T19:40:35.968 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.967+0000 7f475b405640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.985+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47440017f0 con 0x7f475413f710 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.985+0000 7f474bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.985+0000 7f474bfff640 1 -- 172.21.15.107:0/4238416656 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4744001c00 con 0x7f475413f710 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.985+0000 7f474bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.985+0000 7f474bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.985+0000 7f474bfff640 10 monclient: dump: 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:35.985 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:35.986 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:35.986 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:35.986 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:35.986 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:35.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.985+0000 7f475b405640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:35.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.985+0000 7f475b405640 4 set_mon_vals no callback set 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.986+0000 7f475b405640 10 monclient: shutdown 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.986+0000 7f475b405640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.986+0000 7f475b405640 1 -- 172.21.15.107:0/4238416656 >> v1:172.21.15.195:6789/0 conn(0x7f475413f710 legacy=0x7f47541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.986+0000 7f475b405640 1 -- 172.21.15.107:0/4238416656 shutdown_connections 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.986+0000 7f475b405640 1 -- 172.21.15.107:0/4238416656 >> 172.21.15.107:0/4238416656 conn(0x7f47540b7ea0 msgr2=0x7f47540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.986+0000 7f475b405640 1 -- 172.21.15.107:0/4238416656 shutdown_connections 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.986+0000 7f475b405640 1 -- 172.21.15.107:0/4238416656 wait complete. 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.987+0000 7f475b405640 10 monclient: build_initial_monmap 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.987+0000 7f475b405640 10 monclient: monmap: 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:35.988 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.987+0000 7f475b405640 1 librados: starting msgr at 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.987+0000 7f475b405640 1 librados: starting objecter 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.987+0000 7f475b405640 1 Processor -- start 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.987+0000 7f475b405640 1 -- start start 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.987+0000 7f475b405640 1 librados: setting wanted keys 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.987+0000 7f475b405640 1 librados: calling monclient init 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.987+0000 7f475b405640 10 monclient: init 2024-03-25T19:40:35.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.988+0000 7f475b405640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:35.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.988+0000 7f475b405640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:35.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.988+0000 7f475b405640 10 monclient(hunting): picked mon.noname-c con 0x7f47540a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:35.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.988+0000 7f475b405640 10 monclient(hunting): picked mon.noname-b con 0x7f475413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:35.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.988+0000 7f475b405640 10 monclient(hunting): picked mon.noname-a con 0x7f47541430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:35.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.988+0000 7f475b405640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4754138c10 con 0x7f47541430c0 2024-03-25T19:40:35.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.988+0000 7f475b405640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4754138e60 con 0x7f475413f710 2024-03-25T19:40:35.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.988+0000 7f475b405640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4754139140 con 0x7f47540a98f0 2024-03-25T19:40:35.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.988+0000 7f475b405640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:35.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:35.988+0000 7f475b405640 10 monclient(hunting): authenticate will time out at 4828.329246s 2024-03-25T19:40:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.008+0000 7f47599a2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f47541430c0 0x7f47541e34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48092/0 (socket says 172.21.15.107:48092) 2024-03-25T19:40:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.008+0000 7f47599a2640 1 -- 172.21.15.107:0/4002888385 learned_addr learned my addr 172.21.15.107:0/4002888385 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.009+0000 7f4749ffb640 10 client.?.objecter ms_handle_connect 0x7f47541430c0 2024-03-25T19:40:36.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.009+0000 7f4749ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:36.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.012+0000 7f4749ffb640 10 client.?.objecter ms_handle_connect 0x7f475413f710 2024-03-25T19:40:36.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.012+0000 7f4749ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:36.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.016+0000 7f4749ffb640 10 client.?.objecter ms_handle_connect 0x7f47540a98f0 2024-03-25T19:40:36.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.016+0000 7f4749ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894331884 0 0) 0x7f4754138c10 con 0x7f47541430c0 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4738004eb0 con 0x7f47541430c0 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548645044 0 0) 0x7f4754138e60 con 0x7f475413f710 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:40:36.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.026+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4754138c10 con 0x7f475413f710 2024-03-25T19:40:36.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.027+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858599638 0 0) 0x7f4754139140 con 0x7f47540a98f0 2024-03-25T19:40:36.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.027+0000 7f4749ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:36.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.027+0000 7f4749ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:36.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.027+0000 7f4749ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:36.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.027+0000 7f4749ffb640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:40:36.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.027+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4754138e60 con 0x7f47540a98f0 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.038+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2977447597 0 0) 0x7f4738004eb0 con 0x7f47541430c0 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.038+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4754139140 con 0x7f47541430c0 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.038+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f474c001c70 con 0x7f47541430c0 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.038+0000 7f4749ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.038+0000 7f4749ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.038+0000 7f4749ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:36.038 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:36.039 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:36.039 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:36.039 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:36.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.038+0000 7f4749ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:36.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.052+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1471984635 0 0) 0x7f4754138e60 con 0x7f47540a98f0 2024-03-25T19:40:36.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.052+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4738004eb0 con 0x7f47540a98f0 2024-03-25T19:40:36.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.052+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4750001930 con 0x7f47540a98f0 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.052+0000 7f4749ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.052+0000 7f4749ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.052+0000 7f4749ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:36.052 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:36.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 307725623 0 0) 0x7f4738004eb0 con 0x7f47540a98f0 2024-03-25T19:40:36.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 >> v1:172.21.15.195:6789/0 conn(0x7f475413f710 legacy=0x7f4754132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:36.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 >> v1:172.21.15.107:6789/0 conn(0x7f47541430c0 legacy=0x7f47541e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:36.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f4749ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:36.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f4749ffb640 1 monclient: found mon.c 2024-03-25T19:40:36.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f4749ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:36.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f4749ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:40:36.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4754139920 con 0x7f47540a98f0 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f4749ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f4749ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f4749ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f475b405640 5 monclient: authenticate success, global_id 4439 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f475b405640 10 monclient: _renew_subs 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f475b405640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f475b405640 1 -- 172.21.15.107:0/4002888385 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47541382e0 con 0x7f47540a98f0 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f475b405640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f475b405640 10 monclient: _renew_subs 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f475b405640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f475b405640 1 -- 172.21.15.107:0/4002888385 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47540ab8b0 con 0x7f47540a98f0 2024-03-25T19:40:36.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.136+0000 7f475b405640 1 librados: init done 2024-03-25T19:40:36.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.137+0000 7f475b405640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:36.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.137+0000 7f475b405640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:36.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.137+0000 7f475b405640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:40:36.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.137+0000 7f475b405640 1 -- 172.21.15.107:0/4002888385 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4754137a20 con 0x7f47540a98f0 2024-03-25T19:40:36.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.419+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47500035f0 con 0x7f47540a98f0 2024-03-25T19:40:36.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.419+0000 7f4749ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:36.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.419+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4750003900 con 0x7f47540a98f0 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.419+0000 7f4749ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.419+0000 7f4749ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.419+0000 7f4749ffb640 10 monclient: dump: 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:36.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.419+0000 7f474b7fe640 4 set_mon_vals no callback set 2024-03-25T19:40:36.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.420+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f47500198d0 con 0x7f47540a98f0 2024-03-25T19:40:36.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.421+0000 7f47589a0640 10 monclient: get_auth_request con 0x7f473806dfb0 auth_method 0 2024-03-25T19:40:36.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.421+0000 7f4749ffb640 10 client.4439.objecter ms_handle_connect 0x7f473806dfb0 2024-03-25T19:40:36.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.426+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f4750053550 con 0x7f47540a98f0 2024-03-25T19:40:36.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.426+0000 7f4749ffb640 10 client.4439.objecter ms_dispatch 0x7f4754000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:36.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.426+0000 7f4749ffb640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:36.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.426+0000 7f4749ffb640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:36.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.427+0000 7f4749ffb640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-25T19:40:36.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.427+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f475004ee90 con 0x7f47540a98f0 2024-03-25T19:40:36.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.427+0000 7f4749ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:36.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.427+0000 7f4749ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:36.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.674+0000 7f475b405640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:36.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.674+0000 7f475b405640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:36.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.674+0000 7f475b405640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:40:36.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.674+0000 7f475b405640 1 -- 172.21.15.107:0/4002888385 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f475413d010 con 0x7f47540a98f0 2024-03-25T19:40:36.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.714+0000 7f4749ffb640 1 -- 172.21.15.107:0/4002888385 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f4750050f90 con 0x7f47540a98f0 2024-03-25T19:40:36.714 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.714+0000 7f4749ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:36.714 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.714+0000 7f4749ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:36.714 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:40:36.714 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:36.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.716+0000 7f475b405640 10 librados: watch_flush enter 2024-03-25T19:40:36.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.716+0000 7f475b405640 10 librados: watch_flush exit 2024-03-25T19:40:36.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.716+0000 7f475b405640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-25T19:40:36.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.716+0000 7f475b405640 10 client.4439.objecter successfully canceled tick 2024-03-25T19:40:36.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.716+0000 7f475b405640 1 -- 172.21.15.107:0/4002888385 >> v1:172.21.15.195:6800/2984408421 conn(0x7f473806dfb0 legacy=0x7f4738070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:36.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.717+0000 7f475b405640 10 monclient: shutdown 2024-03-25T19:40:36.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.717+0000 7f475b405640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:36.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.717+0000 7f475b405640 1 -- 172.21.15.107:0/4002888385 >> v1:172.21.15.107:6790/0 conn(0x7f47540a98f0 legacy=0x7f4754137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:36.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.717+0000 7f475b405640 1 -- 172.21.15.107:0/4002888385 shutdown_connections 2024-03-25T19:40:36.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.717+0000 7f475b405640 1 -- 172.21.15.107:0/4002888385 >> 172.21.15.107:0/4002888385 conn(0x7f47540b7ea0 msgr2=0x7f4754074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:36.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.717+0000 7f475b405640 1 -- 172.21.15.107:0/4002888385 shutdown_connections 2024-03-25T19:40:36.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.717+0000 7f475b405640 1 -- 172.21.15.107:0/4002888385 wait complete. 2024-03-25T19:40:36.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:36.717+0000 7f475b405640 1 librados: shutdown 2024-03-25T19:40:36.729 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:43.535 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.655+0000 7f026de35640 10 monclient: get_monmap_and_config 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.655+0000 7f026de35640 10 monclient: build_initial_monmap 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.655+0000 7f026de35640 10 monclient: monmap: 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:40:43.656382+0000 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:40:43.656382+0000 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:40:43.655 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:40:43.656 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:43.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 1 Processor -- start 2024-03-25T19:40:43.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 1 -- start start 2024-03-25T19:40:43.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 10 monclient: init 2024-03-25T19:40:43.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:43.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:43.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 10 monclient(hunting): picked mon.noname-a con 0x7f02680b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:43.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 10 monclient(hunting): picked mon.noname-b con 0x7f02680b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:43.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 10 monclient(hunting): picked mon.noname-c con 0x7f026814be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:43.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02680af940 con 0x7f02680b4d60 2024-03-25T19:40:43.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02680b0a70 con 0x7f026814be90 2024-03-25T19:40:43.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02680b0d50 con 0x7f02680b1730 2024-03-25T19:40:43.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:43.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.656+0000 7f026de35640 10 monclient(hunting): authenticate will time out at 4835.997728s 2024-03-25T19:40:43.659 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.660+0000 7f0266ffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f02680b1730 0x7f02680b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45728/0 (socket says 172.21.15.107:45728) 2024-03-25T19:40:43.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.660+0000 7f0266ffd640 1 -- 172.21.15.107:0/2206876948 learned_addr learned my addr 172.21.15.107:0/2206876948 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:43.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.671+0000 7f02667fc640 1 -- 172.21.15.107:0/2206876948 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065617739 0 0) 0x7f02680b0d50 con 0x7f02680b1730 2024-03-25T19:40:43.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.671+0000 7f02667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.671+0000 7f02667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.671+0000 7f02667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.671+0000 7f02667fc640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:40:43.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.671+0000 7f02667fc640 1 -- 172.21.15.107:0/2206876948 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0250003520 con 0x7f02680b1730 2024-03-25T19:40:43.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.681+0000 7f02667fc640 1 -- 172.21.15.107:0/2206876948 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923872368 0 0) 0x7f02680b0a70 con 0x7f026814be90 2024-03-25T19:40:43.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.681+0000 7f02667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.681+0000 7f02667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.681+0000 7f02667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.681+0000 7f02667fc640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:40:43.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.681+0000 7f02667fc640 1 -- 172.21.15.107:0/2206876948 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02680b0d50 con 0x7f026814be90 2024-03-25T19:40:43.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 1 -- 172.21.15.107:0/2206876948 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1937503297 0 0) 0x7f0250003520 con 0x7f02680b1730 2024-03-25T19:40:43.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 1 -- 172.21.15.107:0/2206876948 >> v1:172.21.15.195:6789/0 conn(0x7f026814be90 legacy=0x7f026814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 1 -- 172.21.15.107:0/2206876948 >> v1:172.21.15.107:6789/0 conn(0x7f02680b4d60 legacy=0x7f026813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 1 monclient: found mon.noname-b 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 1 -- 172.21.15.107:0/2206876948 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02680b1030 con 0x7f02680b1730 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 1 -- 172.21.15.107:0/2206876948 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f025c001690 con 0x7f02680b1730 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.686+0000 7f02667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.687+0000 7f02667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.687+0000 7f02667fc640 10 monclient: dump: 2024-03-25T19:40:43.687 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:43.688 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:43.688 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:43.688 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:43.688 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.688 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:43.688 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:43.688 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:43.688 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:43.688 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.687+0000 7f026de35640 5 monclient: authenticate success, global_id 4442 2024-03-25T19:40:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.687+0000 7f026de35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.705+0000 7f02667fc640 1 -- 172.21.15.107:0/2206876948 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f025c0017f0 con 0x7f02680b1730 2024-03-25T19:40:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.705+0000 7f02667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.705+0000 7f02667fc640 1 -- 172.21.15.107:0/2206876948 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f025c001c00 con 0x7f02680b1730 2024-03-25T19:40:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.705+0000 7f026de35640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.705+0000 7f026de35640 4 set_mon_vals no callback set 2024-03-25T19:40:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.705+0000 7f02667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.705+0000 7f02667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.705+0000 7f02667fc640 10 monclient: dump: 2024-03-25T19:40:43.706 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:43.706 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:43.706 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:43.706 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:43.706 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.706 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:43.706 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:43.706 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:43.706 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:43.706 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:43.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.706+0000 7f026de35640 10 monclient: shutdown 2024-03-25T19:40:43.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.706+0000 7f026de35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:43.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.706+0000 7f026de35640 1 -- 172.21.15.107:0/2206876948 >> v1:172.21.15.107:6790/0 conn(0x7f02680b1730 legacy=0x7f02680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.707+0000 7f026de35640 1 -- 172.21.15.107:0/2206876948 shutdown_connections 2024-03-25T19:40:43.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.707+0000 7f026de35640 1 -- 172.21.15.107:0/2206876948 >> 172.21.15.107:0/2206876948 conn(0x7f02680617a0 msgr2=0x7f0268061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:43.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.707+0000 7f026de35640 1 -- 172.21.15.107:0/2206876948 shutdown_connections 2024-03-25T19:40:43.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.707+0000 7f026de35640 1 -- 172.21.15.107:0/2206876948 wait complete. 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.707+0000 7f026de35640 10 monclient: build_initial_monmap 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.707+0000 7f026de35640 10 monclient: monmap: 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.708+0000 7f026de35640 1 librados: starting msgr at 2024-03-25T19:40:43.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.708+0000 7f026de35640 1 librados: starting objecter 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.708+0000 7f026de35640 1 Processor -- start 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.708+0000 7f026de35640 1 -- start start 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.708+0000 7f026de35640 1 librados: setting wanted keys 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.708+0000 7f026de35640 1 librados: calling monclient init 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.708+0000 7f026de35640 10 monclient: init 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.709+0000 7f026de35640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.709+0000 7f026de35640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.709+0000 7f026de35640 10 monclient(hunting): picked mon.noname-a con 0x7f02680b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.709+0000 7f026de35640 10 monclient(hunting): picked mon.noname-c con 0x7f02680b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.709+0000 7f026de35640 10 monclient(hunting): picked mon.noname-b con 0x7f026814be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.709+0000 7f026de35640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02681409c0 con 0x7f02680b1730 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.709+0000 7f026de35640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0268140ee0 con 0x7f026814be90 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.709+0000 7f026de35640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0268141070 con 0x7f02680b4d60 2024-03-25T19:40:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.709+0000 7f026de35640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:43.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.709+0000 7f026de35640 10 monclient(hunting): authenticate will time out at 4836.050397s 2024-03-25T19:40:43.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.717+0000 7f02677fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f02680b1730 0x7f0268131ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58296/0 (socket says 172.21.15.107:58296) 2024-03-25T19:40:43.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.717+0000 7f02677fe640 1 -- 172.21.15.107:0/2377496343 learned_addr learned my addr 172.21.15.107:0/2377496343 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:43.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.719+0000 7f024ffff640 10 client.?.objecter ms_handle_connect 0x7f026814be90 2024-03-25T19:40:43.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.719+0000 7f024ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.722+0000 7f024ffff640 10 client.?.objecter ms_handle_connect 0x7f02680b4d60 2024-03-25T19:40:43.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.722+0000 7f024ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.722+0000 7f024ffff640 10 client.?.objecter ms_handle_connect 0x7f02680b1730 2024-03-25T19:40:43.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.722+0000 7f024ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:43.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.728+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061612592 0 0) 0x7f0268140ee0 con 0x7f026814be90 2024-03-25T19:40:43.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.728+0000 7f024ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.728+0000 7f024ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.728+0000 7f024ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.728+0000 7f024ffff640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:40:43.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.728+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0254004eb0 con 0x7f026814be90 2024-03-25T19:40:43.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.735+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808001766 0 0) 0x7f02681409c0 con 0x7f02680b1730 2024-03-25T19:40:43.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.735+0000 7f024ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.735+0000 7f024ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.735+0000 7f024ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.735+0000 7f024ffff640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:40:43.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.735+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0268140ee0 con 0x7f02680b1730 2024-03-25T19:40:43.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.741+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133725646 0 0) 0x7f0268141070 con 0x7f02680b4d60 2024-03-25T19:40:43.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.741+0000 7f024ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:43.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.741+0000 7f024ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:43.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.741+0000 7f024ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:43.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.741+0000 7f024ffff640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:40:43.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.741+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02681409c0 con 0x7f02680b4d60 2024-03-25T19:40:43.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.754+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3238161059 0 0) 0x7f0254004eb0 con 0x7f026814be90 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0268141070 con 0x7f026814be90 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f0260001bd0 con 0x7f026814be90 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 10 monclient(hunting): dump: 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:43.755 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2159811368 0 0) 0x7f02681409c0 con 0x7f02680b4d60 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0254004eb0 con 0x7f02680b4d60 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f025c001b00 con 0x7f02680b4d60 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.755+0000 7f024ffff640 10 monclient(hunting): dump: 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.756 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:43.757 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:43.757 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:43.757 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:43.757 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:43.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.760+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3101844026 0 0) 0x7f0268140ee0 con 0x7f02680b1730 2024-03-25T19:40:43.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.760+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02681409c0 con 0x7f02680b1730 2024-03-25T19:40:43.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.760+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f02580019b0 con 0x7f02680b1730 2024-03-25T19:40:43.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.760+0000 7f024ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:43.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.760+0000 7f024ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:43.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.760+0000 7f024ffff640 10 monclient(hunting): dump: 2024-03-25T19:40:43.760 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:43.760 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:43.760 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:43.760 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:43.760 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:43.761 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:43.761 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:43.761 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:43.761 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:43.761 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:43.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 196613167 0 0) 0x7f0268141070 con 0x7f026814be90 2024-03-25T19:40:43.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 >> v1:172.21.15.107:6790/0 conn(0x7f02680b4d60 legacy=0x7f02681424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 >> v1:172.21.15.107:6789/0 conn(0x7f02680b1730 legacy=0x7f0268131ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:43.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f024ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:43.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f024ffff640 1 monclient: found mon.b 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f024ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f024ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0268145560 con 0x7f026814be90 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f024ffff640 10 monclient: _finish_auth 0 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f024ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f024ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f026de35640 5 monclient: authenticate success, global_id 4435 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f026de35640 10 monclient: _renew_subs 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f026de35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f026de35640 1 -- 172.21.15.107:0/2377496343 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02681325b0 con 0x7f026814be90 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f026de35640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f026de35640 10 monclient: _renew_subs 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f026de35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f026de35640 1 -- 172.21.15.107:0/2377496343 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02681385d0 con 0x7f026814be90 2024-03-25T19:40:43.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.761+0000 7f026de35640 1 librados: init done 2024-03-25T19:40:43.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.762+0000 7f026de35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:43.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.762+0000 7f026de35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:43.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.762+0000 7f026de35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:43.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:43.762+0000 7f026de35640 1 -- 172.21.15.107:0/2377496343 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0268140b70 con 0x7f026814be90 2024-03-25T19:40:44.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.032+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0260001d30 con 0x7f026814be90 2024-03-25T19:40:44.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.032+0000 7f024ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:44.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.032+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f0260003970 con 0x7f026814be90 2024-03-25T19:40:44.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.032+0000 7f024ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:44.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.032+0000 7f024ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:44.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.032+0000 7f024ffff640 10 monclient: dump: 2024-03-25T19:40:44.033 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:44.033 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:44.033 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:44.033 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:44.033 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:44.033 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:44.033 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:44.033 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:44.033 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:44.033 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:44.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.032+0000 7f0265ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:44.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.033+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f02600199c0 con 0x7f026814be90 2024-03-25T19:40:44.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.034+0000 7f02677fe640 10 monclient: get_auth_request con 0x7f025406e110 auth_method 0 2024-03-25T19:40:44.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.034+0000 7f024ffff640 10 client.4435.objecter ms_handle_connect 0x7f025406e110 2024-03-25T19:40:44.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.036+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f02600536c0 con 0x7f026814be90 2024-03-25T19:40:44.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.036+0000 7f024ffff640 10 client.4435.objecter ms_dispatch 0x7f0268000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:44.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.036+0000 7f024ffff640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:44.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.036+0000 7f024ffff640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:44.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.036+0000 7f024ffff640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T19:40:44.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.039+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f026004f000 con 0x7f026814be90 2024-03-25T19:40:44.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.039+0000 7f024ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:44.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.039+0000 7f024ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:44.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.266+0000 7f026de35640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:44.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.266+0000 7f026de35640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:44.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.266+0000 7f026de35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:44.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.266+0000 7f026de35640 1 -- 172.21.15.107:0/2377496343 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02681421a0 con 0x7f026814be90 2024-03-25T19:40:44.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.296+0000 7f024ffff640 1 -- 172.21.15.107:0/2377496343 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f0260051100 con 0x7f026814be90 2024-03-25T19:40:44.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.296+0000 7f024ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:44.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.296+0000 7f024ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:44.296 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:40:44.296 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:44.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 10 librados: watch_flush enter 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 10 librados: watch_flush exit 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 10 client.4435.objecter successfully canceled tick 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 1 -- 172.21.15.107:0/2377496343 >> v1:172.21.15.195:6800/2984408421 conn(0x7f025406e110 legacy=0x7f02540705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 10 monclient: shutdown 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 1 -- 172.21.15.107:0/2377496343 >> v1:172.21.15.195:6789/0 conn(0x7f026814be90 legacy=0x7f02681e2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 1 -- 172.21.15.107:0/2377496343 shutdown_connections 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 1 -- 172.21.15.107:0/2377496343 >> 172.21.15.107:0/2377496343 conn(0x7f02680617a0 msgr2=0x7f026813c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 1 -- 172.21.15.107:0/2377496343 shutdown_connections 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 1 -- 172.21.15.107:0/2377496343 wait complete. 2024-03-25T19:40:44.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:44.298+0000 7f026de35640 1 librados: shutdown 2024-03-25T19:40:44.310 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:45.091 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:05:15.407 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:45.098 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:05:15.415 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:45.128 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:05:15.444 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:45.161 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:05:15.448 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:45.165 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:05:15.481 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:45.168 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:05:15.455 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:45.169 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:05:15.485 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:45.203 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:05:15.490 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:45.206 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:05:15.522 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:51.117 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:51.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.226+0000 7ff6cb00d640 10 monclient: get_monmap_and_config 2024-03-25T19:40:51.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.226+0000 7ff6cb00d640 10 monclient: build_initial_monmap 2024-03-25T19:40:51.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.226+0000 7ff6cb00d640 10 monclient: monmap: 2024-03-25T19:40:51.226 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:51.226 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:51.226 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:40:51.227449+0000 2024-03-25T19:40:51.226 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:40:51.227449+0000 2024-03-25T19:40:51.226 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:51.227 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:51.227 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:51.227 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:40:51.227 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:40:51.227 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:51.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.227+0000 7ff6cb00d640 1 Processor -- start 2024-03-25T19:40:51.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.227+0000 7ff6cb00d640 1 -- start start 2024-03-25T19:40:51.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.227+0000 7ff6cb00d640 10 monclient: init 2024-03-25T19:40:51.228 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.228+0000 7ff6cb00d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:51.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.228+0000 7ff6cb00d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:51.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.228+0000 7ff6cb00d640 10 monclient(hunting): picked mon.noname-c con 0x7ff6c40a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:51.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.229+0000 7ff6cb00d640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c413f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:51.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.229+0000 7ff6cb00d640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c41430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:51.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.229+0000 7ff6cb00d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c4142520 con 0x7ff6c41430c0 2024-03-25T19:40:51.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.229+0000 7ff6cb00d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c40b28a0 con 0x7ff6c40a98f0 2024-03-25T19:40:51.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.229+0000 7ff6cb00d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c40b2a30 con 0x7ff6c413f710 2024-03-25T19:40:51.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.229+0000 7ff6cb00d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:51.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.229+0000 7ff6cb00d640 10 monclient(hunting): authenticate will time out at 4843.570021s 2024-03-25T19:40:51.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.237+0000 7ff6c3fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff6c413f710 0x7ff6c41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:57164/0 (socket says 172.21.15.107:57164) 2024-03-25T19:40:51.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.237+0000 7ff6c3fff640 1 -- 172.21.15.107:0/621650161 learned_addr learned my addr 172.21.15.107:0/621650161 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:51.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.252+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870917520 0 0) 0x7ff6c4142520 con 0x7ff6c41430c0 2024-03-25T19:40:51.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.252+0000 7ff6c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.252+0000 7ff6c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.252+0000 7ff6c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.252+0000 7ff6c37fe640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:40:51.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.252+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6ac003520 con 0x7ff6c41430c0 2024-03-25T19:40:51.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.257+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973978899 0 0) 0x7ff6c40b28a0 con 0x7ff6c40a98f0 2024-03-25T19:40:51.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.257+0000 7ff6c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.257+0000 7ff6c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.257+0000 7ff6c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.257+0000 7ff6c37fe640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:40:51.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.257+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c4142520 con 0x7ff6c40a98f0 2024-03-25T19:40:51.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.260+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840105435 0 0) 0x7ff6c40b2a30 con 0x7ff6c413f710 2024-03-25T19:40:51.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.260+0000 7ff6c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.260+0000 7ff6c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.260+0000 7ff6c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.260+0000 7ff6c37fe640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:40:51.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.260+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c40b28a0 con 0x7ff6c413f710 2024-03-25T19:40:51.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3512358964 0 0) 0x7ff6ac003520 con 0x7ff6c41430c0 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 >> v1:172.21.15.107:6790/0 conn(0x7ff6c413f710 legacy=0x7ff6c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 >> v1:172.21.15.195:6789/0 conn(0x7ff6c40a98f0 legacy=0x7ff6c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 1 monclient: found mon.noname-a 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c40b44c0 con 0x7ff6c41430c0 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff6bc001a10 con 0x7ff6c41430c0 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6cb00d640 5 monclient: authenticate success, global_id 4446 2024-03-25T19:40:51.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.270+0000 7ff6cb00d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.271+0000 7ff6c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.271+0000 7ff6c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.271+0000 7ff6c37fe640 10 monclient: dump: 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:51.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.271+0000 7ff6cb00d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:51.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.284+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6bc001b70 con 0x7ff6c41430c0 2024-03-25T19:40:51.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.284+0000 7ff6c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:51.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.285+0000 7ff6cb00d640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.285+0000 7ff6c37fe640 1 -- 172.21.15.107:0/621650161 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff6bc001f80 con 0x7ff6c41430c0 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.285+0000 7ff6cb00d640 4 set_mon_vals no callback set 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.285+0000 7ff6c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.285+0000 7ff6c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.285+0000 7ff6c37fe640 10 monclient: dump: 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:51.285 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.285+0000 7ff6cb00d640 10 monclient: shutdown 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.285+0000 7ff6cb00d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.285+0000 7ff6cb00d640 1 -- 172.21.15.107:0/621650161 >> v1:172.21.15.107:6789/0 conn(0x7ff6c41430c0 legacy=0x7ff6c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.285+0000 7ff6cb00d640 1 -- 172.21.15.107:0/621650161 shutdown_connections 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.285+0000 7ff6cb00d640 1 -- 172.21.15.107:0/621650161 >> 172.21.15.107:0/621650161 conn(0x7ff6c40b7ea0 msgr2=0x7ff6c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.286+0000 7ff6cb00d640 1 -- 172.21.15.107:0/621650161 shutdown_connections 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.286+0000 7ff6cb00d640 1 -- 172.21.15.107:0/621650161 wait complete. 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.286+0000 7ff6cb00d640 10 monclient: build_initial_monmap 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.286+0000 7ff6cb00d640 10 monclient: monmap: 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:40:51.288 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.287+0000 7ff6cb00d640 1 librados: starting msgr at 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.287+0000 7ff6cb00d640 1 librados: starting objecter 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.287+0000 7ff6cb00d640 1 Processor -- start 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.287+0000 7ff6cb00d640 1 -- start start 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.287+0000 7ff6cb00d640 1 librados: setting wanted keys 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.287+0000 7ff6cb00d640 1 librados: calling monclient init 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.287+0000 7ff6cb00d640 10 monclient: init 2024-03-25T19:40:51.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.288+0000 7ff6cb00d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:51.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.288+0000 7ff6cb00d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:51.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.288+0000 7ff6cb00d640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c40a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:51.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.288+0000 7ff6cb00d640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c413f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:51.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.288+0000 7ff6cb00d640 10 monclient(hunting): picked mon.noname-c con 0x7ff6c41430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:51.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.288+0000 7ff6cb00d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c4136a50 con 0x7ff6c413f710 2024-03-25T19:40:51.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.288+0000 7ff6cb00d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c40b79c0 con 0x7ff6c40a98f0 2024-03-25T19:40:51.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.288+0000 7ff6cb00d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c40b7ca0 con 0x7ff6c41430c0 2024-03-25T19:40:51.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.288+0000 7ff6cb00d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:51.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.288+0000 7ff6cb00d640 10 monclient(hunting): authenticate will time out at 4843.628988s 2024-03-25T19:40:51.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.300+0000 7ff6c95aa640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff6c41430c0 0x7ff6c41e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:57208/0 (socket says 172.21.15.107:57208) 2024-03-25T19:40:51.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.300+0000 7ff6c95aa640 1 -- 172.21.15.107:0/847292641 learned_addr learned my addr 172.21.15.107:0/847292641 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:51.301 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.302+0000 7ff6c17fa640 10 client.?.objecter ms_handle_connect 0x7ff6c40a98f0 2024-03-25T19:40:51.301 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.302+0000 7ff6c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:51.301 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.302+0000 7ff6c17fa640 10 client.?.objecter ms_handle_connect 0x7ff6c41430c0 2024-03-25T19:40:51.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.302+0000 7ff6c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:51.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.311+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065090952 0 0) 0x7ff6c40b79c0 con 0x7ff6c40a98f0 2024-03-25T19:40:51.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.311+0000 7ff6c17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.311+0000 7ff6c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.311+0000 7ff6c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.311+0000 7ff6c17fa640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:40:51.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.311+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b0004eb0 con 0x7ff6c40a98f0 2024-03-25T19:40:51.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.314+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810345523 0 0) 0x7ff6c40b7ca0 con 0x7ff6c41430c0 2024-03-25T19:40:51.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.314+0000 7ff6c17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.314+0000 7ff6c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.314+0000 7ff6c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.314+0000 7ff6c17fa640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:40:51.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.314+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c40b79c0 con 0x7ff6c41430c0 2024-03-25T19:40:51.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.317+0000 7ff6c17fa640 10 client.?.objecter ms_handle_connect 0x7ff6c413f710 2024-03-25T19:40:51.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.317+0000 7ff6c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:51.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.325+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431591928 0 0) 0x7ff6c4136a50 con 0x7ff6c413f710 2024-03-25T19:40:51.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.325+0000 7ff6c17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.325+0000 7ff6c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.325+0000 7ff6c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.325+0000 7ff6c17fa640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:40:51.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.325+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c40b7ca0 con 0x7ff6c413f710 2024-03-25T19:40:51.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.332+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474491062 0 0) 0x7ff6b0004eb0 con 0x7ff6c40a98f0 2024-03-25T19:40:51.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.332+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6c4136a50 con 0x7ff6c40a98f0 2024-03-25T19:40:51.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.332+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff6b4001920 con 0x7ff6c40a98f0 2024-03-25T19:40:51.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.332+0000 7ff6c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.332+0000 7ff6c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:51.332 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.332+0000 7ff6c17fa640 10 monclient(hunting): dump: 2024-03-25T19:40:51.332 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:51.332 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:51.333 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:51.333 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:51.333 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.333 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:51.333 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:51.333 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:51.333 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:51.333 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:51.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.332+0000 7ff6c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:51.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.333+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 754595381 0 0) 0x7ff6c40b79c0 con 0x7ff6c41430c0 2024-03-25T19:40:51.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.333+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6b0004eb0 con 0x7ff6c41430c0 2024-03-25T19:40:51.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.334+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff6bc001ec0 con 0x7ff6c41430c0 2024-03-25T19:40:51.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.334+0000 7ff6c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.334+0000 7ff6c17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:51.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.334+0000 7ff6c17fa640 10 monclient(hunting): dump: 2024-03-25T19:40:51.334 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:51.334 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:51.334 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:51.335 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:51.335 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.335 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:51.335 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:51.335 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:51.335 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:51.335 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:51.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2662456349 0 0) 0x7ff6c4136a50 con 0x7ff6c40a98f0 2024-03-25T19:40:51.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 >> v1:172.21.15.107:6790/0 conn(0x7ff6c41430c0 legacy=0x7ff6c41e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 >> v1:172.21.15.107:6789/0 conn(0x7ff6c413f710 legacy=0x7ff6c4135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6c17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6c17fa640 1 monclient: found mon.b 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6c17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c4136ca0 con 0x7ff6c40a98f0 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6c17fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6c17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6cb00d640 5 monclient: authenticate success, global_id 4441 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6cb00d640 10 monclient: _renew_subs 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6cb00d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6cb00d640 1 -- 172.21.15.107:0/847292641 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c4136f80 con 0x7ff6c40a98f0 2024-03-25T19:40:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.346+0000 7ff6cb00d640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.347+0000 7ff6cb00d640 10 monclient: _renew_subs 2024-03-25T19:40:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.347+0000 7ff6cb00d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.347+0000 7ff6cb00d640 1 -- 172.21.15.107:0/847292641 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c41360e0 con 0x7ff6c40a98f0 2024-03-25T19:40:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.347+0000 7ff6cb00d640 1 librados: init done 2024-03-25T19:40:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.348+0000 7ff6cb00d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.348+0000 7ff6cb00d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.348+0000 7ff6cb00d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.348+0000 7ff6cb00d640 1 -- 172.21.15.107:0/847292641 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c40627b0 con 0x7ff6c40a98f0 2024-03-25T19:40:51.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.630+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6b4003610 con 0x7ff6c40a98f0 2024-03-25T19:40:51.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.630+0000 7ff6c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:51.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.630+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff6b4003920 con 0x7ff6c40a98f0 2024-03-25T19:40:51.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.630+0000 7ff6c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.630+0000 7ff6c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.630+0000 7ff6c17fa640 10 monclient: dump: 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:51.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.630+0000 7ff6c2ffd640 4 set_mon_vals no callback set 2024-03-25T19:40:51.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.631+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7ff6b40198f0 con 0x7ff6c40a98f0 2024-03-25T19:40:51.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.632+0000 7ff6c3fff640 10 monclient: get_auth_request con 0x7ff6b006dec0 auth_method 0 2024-03-25T19:40:51.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.632+0000 7ff6c17fa640 10 client.4441.objecter ms_handle_connect 0x7ff6b006dec0 2024-03-25T19:40:51.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.639+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7ff6b40535f0 con 0x7ff6c40a98f0 2024-03-25T19:40:51.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.639+0000 7ff6c17fa640 10 client.4441.objecter ms_dispatch 0x7ff6c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:51.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.639+0000 7ff6c17fa640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:51.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.639+0000 7ff6c17fa640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:51.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.639+0000 7ff6c17fa640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-25T19:40:51.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.639+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6b404ef30 con 0x7ff6c40a98f0 2024-03-25T19:40:51.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.639+0000 7ff6c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:51.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.639+0000 7ff6c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:51.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.885+0000 7ff6cb00d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:51.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.885+0000 7ff6cb00d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:51.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.885+0000 7ff6cb00d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:51.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.885+0000 7ff6cb00d640 1 -- 172.21.15.107:0/847292641 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6c413e290 con 0x7ff6c40a98f0 2024-03-25T19:40:51.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.911+0000 7ff6c17fa640 1 -- 172.21.15.107:0/847292641 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7ff6b4051030 con 0x7ff6c40a98f0 2024-03-25T19:40:51.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.911+0000 7ff6c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:51.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.911+0000 7ff6c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:51.911 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:40:51.911 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:51.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.913+0000 7ff6cb00d640 10 librados: watch_flush enter 2024-03-25T19:40:51.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.913+0000 7ff6cb00d640 10 librados: watch_flush exit 2024-03-25T19:40:51.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.913+0000 7ff6cb00d640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-25T19:40:51.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.914+0000 7ff6cb00d640 10 client.4441.objecter successfully canceled tick 2024-03-25T19:40:51.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.914+0000 7ff6cb00d640 1 -- 172.21.15.107:0/847292641 >> v1:172.21.15.195:6800/2984408421 conn(0x7ff6b006dec0 legacy=0x7ff6b0070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.914+0000 7ff6cb00d640 10 monclient: shutdown 2024-03-25T19:40:51.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.914+0000 7ff6cb00d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:51.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.914+0000 7ff6cb00d640 1 -- 172.21.15.107:0/847292641 >> v1:172.21.15.195:6789/0 conn(0x7ff6c40a98f0 legacy=0x7ff6c4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.914+0000 7ff6cb00d640 1 -- 172.21.15.107:0/847292641 shutdown_connections 2024-03-25T19:40:51.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.914+0000 7ff6cb00d640 1 -- 172.21.15.107:0/847292641 >> 172.21.15.107:0/847292641 conn(0x7ff6c40b7ea0 msgr2=0x7ff6c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:51.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.914+0000 7ff6cb00d640 1 -- 172.21.15.107:0/847292641 shutdown_connections 2024-03-25T19:40:51.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.914+0000 7ff6cb00d640 1 -- 172.21.15.107:0/847292641 wait complete. 2024-03-25T19:40:51.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:51.914+0000 7ff6cb00d640 1 librados: shutdown 2024-03-25T19:40:51.926 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:58.732 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:58.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.849+0000 7f262f97b640 10 monclient: get_monmap_and_config 2024-03-25T19:40:58.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.849+0000 7f262f97b640 10 monclient: build_initial_monmap 2024-03-25T19:40:58.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.849+0000 7f262f97b640 10 monclient: monmap: 2024-03-25T19:40:58.849 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:58.849 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:58.849 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:40:58.850288+0000 2024-03-25T19:40:58.849 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:40:58.850288+0000 2024-03-25T19:40:58.849 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:58.849 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:58.850 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:58.850 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:40:58.850 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:40:58.850 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:58.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.850+0000 7f262f97b640 1 Processor -- start 2024-03-25T19:40:58.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.850+0000 7f262f97b640 1 -- start start 2024-03-25T19:40:58.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.850+0000 7f262f97b640 10 monclient: init 2024-03-25T19:40:58.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.851+0000 7f262f97b640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:58.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.851+0000 7f262f97b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:58.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.851+0000 7f262f97b640 10 monclient(hunting): picked mon.noname-b con 0x7f26280a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:58.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.851+0000 7f262f97b640 10 monclient(hunting): picked mon.noname-a con 0x7f262813f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:58.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.851+0000 7f262f97b640 10 monclient(hunting): picked mon.noname-c con 0x7f26281430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:58.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.851+0000 7f262f97b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2628142520 con 0x7f262813f710 2024-03-25T19:40:58.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.851+0000 7f262f97b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26280b28a0 con 0x7f26281430c0 2024-03-25T19:40:58.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.851+0000 7f262f97b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26280b2a30 con 0x7f26280a98f0 2024-03-25T19:40:58.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.851+0000 7f262f97b640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:58.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.851+0000 7f262f97b640 10 monclient(hunting): authenticate will time out at 4851.192211s 2024-03-25T19:40:58.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.859+0000 7f262d717640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f26280a98f0 0x7f26280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:33758/0 (socket says 172.21.15.107:33758) 2024-03-25T19:40:58.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.859+0000 7f262d717640 1 -- 172.21.15.107:0/435452183 learned_addr learned my addr 172.21.15.107:0/435452183 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:58.867 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.867+0000 7f261ffff640 1 -- 172.21.15.107:0/435452183 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093035384 0 0) 0x7f26280b2a30 con 0x7f26280a98f0 2024-03-25T19:40:58.867 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.868+0000 7f261ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:58.867 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.868+0000 7f261ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:58.867 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.868+0000 7f261ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:58.867 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.868+0000 7f261ffff640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:40:58.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.868+0000 7f261ffff640 1 -- 172.21.15.107:0/435452183 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2614003520 con 0x7f26280a98f0 2024-03-25T19:40:58.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.875+0000 7f261ffff640 1 -- 172.21.15.107:0/435452183 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866261168 0 0) 0x7f26280b28a0 con 0x7f26281430c0 2024-03-25T19:40:58.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.875+0000 7f261ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:58.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.875+0000 7f261ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:58.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.875+0000 7f261ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:58.875 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.875+0000 7f261ffff640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:40:58.875 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.875+0000 7f261ffff640 1 -- 172.21.15.107:0/435452183 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26280b2a30 con 0x7f26281430c0 2024-03-25T19:40:58.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 1 -- 172.21.15.107:0/435452183 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3383518298 0 0) 0x7f26280b2a30 con 0x7f26281430c0 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 1 -- 172.21.15.107:0/435452183 >> v1:172.21.15.107:6790/0 conn(0x7f26280a98f0 legacy=0x7f26280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 1 -- 172.21.15.107:0/435452183 >> v1:172.21.15.107:6789/0 conn(0x7f262813f710 legacy=0x7f26281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 1 monclient: found mon.noname-c 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 1 -- 172.21.15.107:0/435452183 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26280b44c0 con 0x7f26281430c0 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 10 monclient: _finish_auth 0 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f262f97b640 5 monclient: authenticate success, global_id 4444 2024-03-25T19:40:58.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 1 -- 172.21.15.107:0/435452183 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f26200019f0 con 0x7f26281430c0 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f262f97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f261ffff640 10 monclient: dump: 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:58.891 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:58.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.890+0000 7f262f97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.904+0000 7f261ffff640 1 -- 172.21.15.107:0/435452183 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2620001b50 con 0x7f26281430c0 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.904+0000 7f261ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.904+0000 7f261ffff640 1 -- 172.21.15.107:0/435452183 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2620001f20 con 0x7f26281430c0 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.904+0000 7f262f97b640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.904+0000 7f262f97b640 4 set_mon_vals no callback set 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.904+0000 7f261ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.904+0000 7f261ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.904+0000 7f261ffff640 10 monclient: dump: 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:58.904 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:58.905 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:58.905 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:58.905 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:58.905 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:58.905 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:58.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.905+0000 7f262f97b640 10 monclient: shutdown 2024-03-25T19:40:58.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.905+0000 7f262f97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:58.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.905+0000 7f262f97b640 1 -- 172.21.15.107:0/435452183 >> v1:172.21.15.195:6789/0 conn(0x7f26281430c0 legacy=0x7f262814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:58.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.905+0000 7f262f97b640 1 -- 172.21.15.107:0/435452183 shutdown_connections 2024-03-25T19:40:58.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.905+0000 7f262f97b640 1 -- 172.21.15.107:0/435452183 >> 172.21.15.107:0/435452183 conn(0x7f26280b7ea0 msgr2=0x7f26280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:58.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.905+0000 7f262f97b640 1 -- 172.21.15.107:0/435452183 shutdown_connections 2024-03-25T19:40:58.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.905+0000 7f262f97b640 1 -- 172.21.15.107:0/435452183 wait complete. 2024-03-25T19:40:58.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.906+0000 7f262f97b640 10 monclient: build_initial_monmap 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.906+0000 7f262f97b640 10 monclient: monmap: 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.906+0000 7f262f97b640 1 librados: starting msgr at 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.906+0000 7f262f97b640 1 librados: starting objecter 2024-03-25T19:40:58.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.906+0000 7f262f97b640 1 Processor -- start 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.906+0000 7f262f97b640 1 -- start start 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.906+0000 7f262f97b640 1 librados: setting wanted keys 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.906+0000 7f262f97b640 1 librados: calling monclient init 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.906+0000 7f262f97b640 10 monclient: init 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.907+0000 7f262f97b640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.907+0000 7f262f97b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.907+0000 7f262f97b640 10 monclient(hunting): picked mon.noname-a con 0x7f26280a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.907+0000 7f262f97b640 10 monclient(hunting): picked mon.noname-b con 0x7f262813f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.907+0000 7f262f97b640 10 monclient(hunting): picked mon.noname-c con 0x7f26281430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.907+0000 7f262f97b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26280ae940 con 0x7f26280a98f0 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.907+0000 7f262f97b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26280b0870 con 0x7f262813f710 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.907+0000 7f262f97b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26280b0b50 con 0x7f26281430c0 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.907+0000 7f262f97b640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:58.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.907+0000 7f262f97b640 10 monclient(hunting): authenticate will time out at 4851.248079s 2024-03-25T19:40:58.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.911+0000 7f262d717640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f26280a98f0 0x7f26280af0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40456/0 (socket says 172.21.15.107:40456) 2024-03-25T19:40:58.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.911+0000 7f262d717640 1 -- 172.21.15.107:0/2702004655 learned_addr learned my addr 172.21.15.107:0/2702004655 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:40:58.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.912+0000 7f261dffb640 10 client.?.objecter ms_handle_connect 0x7f26280a98f0 2024-03-25T19:40:58.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.912+0000 7f261dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:58.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.922+0000 7f261dffb640 10 client.?.objecter ms_handle_connect 0x7f262813f710 2024-03-25T19:40:58.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.922+0000 7f261dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:58.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.924+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249833448 0 0) 0x7f26280ae940 con 0x7f26280a98f0 2024-03-25T19:40:58.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.924+0000 7f261dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:58.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.924+0000 7f261dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:58.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.924+0000 7f261dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:58.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.924+0000 7f261dffb640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:40:58.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.924+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f260c004eb0 con 0x7f26280a98f0 2024-03-25T19:40:58.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.927+0000 7f261dffb640 10 client.?.objecter ms_handle_connect 0x7f26281430c0 2024-03-25T19:40:58.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.927+0000 7f261dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:58.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.938+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902717148 0 0) 0x7f26280b0870 con 0x7f262813f710 2024-03-25T19:40:58.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.938+0000 7f261dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:58.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.938+0000 7f261dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:58.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.938+0000 7f261dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:58.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.938+0000 7f261dffb640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:40:58.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.938+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26280ae940 con 0x7f262813f710 2024-03-25T19:40:58.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.947+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745350252 0 0) 0x7f26280b0b50 con 0x7f26281430c0 2024-03-25T19:40:58.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.947+0000 7f261dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:58.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.947+0000 7f261dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:58.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.947+0000 7f261dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:58.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.948+0000 7f261dffb640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:40:58.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.948+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26280b0870 con 0x7f26281430c0 2024-03-25T19:40:58.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.952+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 659811010 0 0) 0x7f260c004eb0 con 0x7f26280a98f0 2024-03-25T19:40:58.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.952+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26280b0b50 con 0x7f26280a98f0 2024-03-25T19:40:58.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.952+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2624001950 con 0x7f26280a98f0 2024-03-25T19:40:58.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.952+0000 7f261dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:58.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.952+0000 7f261dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:58.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.952+0000 7f261dffb640 10 monclient(hunting): dump: 2024-03-25T19:40:58.952 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:58.952 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:58.952 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:58.952 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:58.952 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:58.953 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:58.953 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:58.953 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:58.953 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:58.953 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:58.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.952+0000 7f261dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.957+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1861644022 0 0) 0x7f26280ae940 con 0x7f262813f710 2024-03-25T19:40:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.957+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f260c004eb0 con 0x7f262813f710 2024-03-25T19:40:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.957+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f26180018b0 con 0x7f262813f710 2024-03-25T19:40:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.957+0000 7f261dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.957+0000 7f261dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.957+0000 7f261dffb640 10 monclient(hunting): dump: 2024-03-25T19:40:58.957 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:58.958 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:58.958 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:58.958 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:58.958 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:58.958 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:58.958 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:58.958 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:58.958 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:58.958 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:58.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.968+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1591537979 0 0) 0x7f260c004eb0 con 0x7f262813f710 2024-03-25T19:40:58.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.968+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 >> v1:172.21.15.107:6790/0 conn(0x7f26281430c0 legacy=0x7f26281d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:58.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.968+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 >> v1:172.21.15.107:6789/0 conn(0x7f26280a98f0 legacy=0x7f26280af0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:58.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.968+0000 7f261dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:58.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.968+0000 7f261dffb640 1 monclient: found mon.b 2024-03-25T19:40:58.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.968+0000 7f261dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:58.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.968+0000 7f261dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:58.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.968+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26280aeb70 con 0x7f262813f710 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.968+0000 7f261dffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.968+0000 7f261dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.968+0000 7f261dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.969+0000 7f262f97b640 5 monclient: authenticate success, global_id 4447 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.969+0000 7f262f97b640 10 monclient: _renew_subs 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.969+0000 7f262f97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.969+0000 7f262f97b640 1 -- 172.21.15.107:0/2702004655 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26281d82b0 con 0x7f262813f710 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.969+0000 7f262f97b640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.969+0000 7f262f97b640 10 monclient: _renew_subs 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.969+0000 7f262f97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.969+0000 7f262f97b640 1 -- 172.21.15.107:0/2702004655 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26280af9d0 con 0x7f262813f710 2024-03-25T19:40:58.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.969+0000 7f262f97b640 1 librados: init done 2024-03-25T19:40:58.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.970+0000 7f262f97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:58.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.970+0000 7f262f97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:58.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.970+0000 7f262f97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:58.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:58.970+0000 7f262f97b640 1 -- 172.21.15.107:0/2702004655 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262814f2d0 con 0x7f262813f710 2024-03-25T19:40:59.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.246+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26180035f0 con 0x7f262813f710 2024-03-25T19:40:59.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.246+0000 7f261dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:59.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.246+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2618003940 con 0x7f262813f710 2024-03-25T19:40:59.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.246+0000 7f261dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.246+0000 7f261dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:59.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.246+0000 7f261dffb640 10 monclient: dump: 2024-03-25T19:40:59.246 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:40:59.247 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:40:59.247 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:59.247 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:40:59.247 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.247 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:40:59.247 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:40:59.247 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:40:59.247 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:40:59.247 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:40:59.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.246+0000 7f261f7fe640 4 set_mon_vals no callback set 2024-03-25T19:40:59.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.247+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f2618003b60 con 0x7f262813f710 2024-03-25T19:40:59.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.248+0000 7f262d717640 10 monclient: get_auth_request con 0x7f260c06dfb0 auth_method 0 2024-03-25T19:40:59.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.248+0000 7f261dffb640 10 client.4447.objecter ms_handle_connect 0x7f260c06dfb0 2024-03-25T19:40:59.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.253+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f2618053590 con 0x7f262813f710 2024-03-25T19:40:59.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.253+0000 7f261dffb640 10 client.4447.objecter ms_dispatch 0x7f2628000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:59.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.253+0000 7f261dffb640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:59.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.253+0000 7f261dffb640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:59.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.253+0000 7f261dffb640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-25T19:40:59.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.254+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f261804f0a0 con 0x7f262813f710 2024-03-25T19:40:59.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.254+0000 7f261dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.254+0000 7f261dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:59.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.482+0000 7f262f97b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:59.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.482+0000 7f262f97b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:59.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.482+0000 7f262f97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:40:59.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.482+0000 7f262f97b640 1 -- 172.21.15.107:0/2702004655 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26280aed00 con 0x7f262813f710 2024-03-25T19:40:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.507+0000 7f261dffb640 1 -- 172.21.15.107:0/2702004655 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f2618050fd0 con 0x7f262813f710 2024-03-25T19:40:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.507+0000 7f261dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.507+0000 7f261dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:59.507 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:40:59.507 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 10 librados: watch_flush enter 2024-03-25T19:40:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 10 librados: watch_flush exit 2024-03-25T19:40:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-25T19:40:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 10 client.4447.objecter successfully canceled tick 2024-03-25T19:40:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 1 -- 172.21.15.107:0/2702004655 >> v1:172.21.15.195:6800/2984408421 conn(0x7f260c06dfb0 legacy=0x7f260c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 10 monclient: shutdown 2024-03-25T19:40:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 1 -- 172.21.15.107:0/2702004655 >> v1:172.21.15.195:6789/0 conn(0x7f262813f710 legacy=0x7f26281d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 1 -- 172.21.15.107:0/2702004655 shutdown_connections 2024-03-25T19:40:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 1 -- 172.21.15.107:0/2702004655 >> 172.21.15.107:0/2702004655 conn(0x7f26280b7ea0 msgr2=0x7f2628074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 1 -- 172.21.15.107:0/2702004655 shutdown_connections 2024-03-25T19:40:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 1 -- 172.21.15.107:0/2702004655 wait complete. 2024-03-25T19:40:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:40:59.510+0000 7f262f97b640 1 librados: shutdown 2024-03-25T19:40:59.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:05.185 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:05:35.501 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:05.190 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:05:35.507 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:05.225 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:05:35.541 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:05.242 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:05:35.529 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:05.244 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:05:35.560 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:05.248 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:05:35.535 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:05.253 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:05:35.569 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:05.272 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:05:35.559 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:05.280 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:05:35.596 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:06.329 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:06.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.428+0000 7ff27658c640 10 monclient: get_monmap_and_config 2024-03-25T19:41:06.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.428+0000 7ff27658c640 10 monclient: build_initial_monmap 2024-03-25T19:41:06.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.428+0000 7ff27658c640 10 monclient: monmap: 2024-03-25T19:41:06.428 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:06.428 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:06.428 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:41:06.429642+0000 2024-03-25T19:41:06.429 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:41:06.429642+0000 2024-03-25T19:41:06.429 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:06.429 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:06.429 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:06.429 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:41:06.429 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:41:06.429 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:06.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.429+0000 7ff27658c640 1 Processor -- start 2024-03-25T19:41:06.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.429+0000 7ff27658c640 1 -- start start 2024-03-25T19:41:06.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.429+0000 7ff27658c640 10 monclient: init 2024-03-25T19:41:06.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.430+0000 7ff27658c640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:06.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.430+0000 7ff27658c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:06.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.430+0000 7ff27658c640 10 monclient(hunting): picked mon.noname-c con 0x7ff2700a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:06.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.430+0000 7ff27658c640 10 monclient(hunting): picked mon.noname-b con 0x7ff27013f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:06.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.430+0000 7ff27658c640 10 monclient(hunting): picked mon.noname-a con 0x7ff2701430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:06.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.430+0000 7ff27658c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff270142520 con 0x7ff2701430c0 2024-03-25T19:41:06.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.430+0000 7ff27658c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2700b28a0 con 0x7ff2700a98f0 2024-03-25T19:41:06.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.430+0000 7ff27658c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2700b2a30 con 0x7ff27013f710 2024-03-25T19:41:06.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.430+0000 7ff27658c640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:06.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.430+0000 7ff27658c640 10 monclient(hunting): authenticate will time out at 4858.770962s 2024-03-25T19:41:06.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.433+0000 7ff26ffff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff2700a98f0 0x7ff2700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:55748/0 (socket says 172.21.15.107:55748) 2024-03-25T19:41:06.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.433+0000 7ff26ffff640 1 -- 172.21.15.107:0/1580373642 learned_addr learned my addr 172.21.15.107:0/1580373642 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:06.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.453+0000 7ff26effd640 1 -- 172.21.15.107:0/1580373642 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348290554 0 0) 0x7ff2700b2a30 con 0x7ff27013f710 2024-03-25T19:41:06.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.453+0000 7ff26effd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:06.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.453+0000 7ff26effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:06.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.453+0000 7ff26effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:06.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.453+0000 7ff26effd640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:41:06.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.453+0000 7ff26effd640 1 -- 172.21.15.107:0/1580373642 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff258003520 con 0x7ff27013f710 2024-03-25T19:41:06.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.459+0000 7ff26effd640 1 -- 172.21.15.107:0/1580373642 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103375587 0 0) 0x7ff270142520 con 0x7ff2701430c0 2024-03-25T19:41:06.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.459+0000 7ff26effd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:06.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.459+0000 7ff26effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:06.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.459+0000 7ff26effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:06.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.459+0000 7ff26effd640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:41:06.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.459+0000 7ff26effd640 1 -- 172.21.15.107:0/1580373642 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2700b2a30 con 0x7ff2701430c0 2024-03-25T19:41:06.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 1 -- 172.21.15.107:0/1580373642 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4223368649 0 0) 0x7ff258003520 con 0x7ff27013f710 2024-03-25T19:41:06.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 1 -- 172.21.15.107:0/1580373642 >> v1:172.21.15.195:6789/0 conn(0x7ff2700a98f0 legacy=0x7ff2700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 1 -- 172.21.15.107:0/1580373642 >> v1:172.21.15.107:6789/0 conn(0x7ff2701430c0 legacy=0x7ff27014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 1 monclient: found mon.noname-b 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 1 -- 172.21.15.107:0/1580373642 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2700b44c0 con 0x7ff27013f710 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 10 monclient: _finish_auth 0 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 1 -- 172.21.15.107:0/1580373642 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff264001690 con 0x7ff27013f710 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff27658c640 5 monclient: authenticate success, global_id 4460 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff27658c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:06.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff26effd640 10 monclient: dump: 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:06.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.468+0000 7ff27658c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:06.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.493+0000 7ff26effd640 1 -- 172.21.15.107:0/1580373642 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2640017f0 con 0x7ff27013f710 2024-03-25T19:41:06.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.493+0000 7ff26effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.493+0000 7ff26effd640 1 -- 172.21.15.107:0/1580373642 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff264001c00 con 0x7ff27013f710 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.493+0000 7ff27658c640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.493+0000 7ff27658c640 4 set_mon_vals no callback set 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.493+0000 7ff26effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.493+0000 7ff26effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.493+0000 7ff26effd640 10 monclient: dump: 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:06.493 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:06.494 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:06.494 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:06.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.494+0000 7ff27658c640 10 monclient: shutdown 2024-03-25T19:41:06.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.494+0000 7ff27658c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:06.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.494+0000 7ff27658c640 1 -- 172.21.15.107:0/1580373642 >> v1:172.21.15.107:6790/0 conn(0x7ff27013f710 legacy=0x7ff2701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:06.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.494+0000 7ff27658c640 1 -- 172.21.15.107:0/1580373642 shutdown_connections 2024-03-25T19:41:06.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.494+0000 7ff27658c640 1 -- 172.21.15.107:0/1580373642 >> 172.21.15.107:0/1580373642 conn(0x7ff2700b7ea0 msgr2=0x7ff2700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:06.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.494+0000 7ff27658c640 1 -- 172.21.15.107:0/1580373642 shutdown_connections 2024-03-25T19:41:06.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.494+0000 7ff27658c640 1 -- 172.21.15.107:0/1580373642 wait complete. 2024-03-25T19:41:06.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.495+0000 7ff27658c640 10 monclient: build_initial_monmap 2024-03-25T19:41:06.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.495+0000 7ff27658c640 10 monclient: monmap: 2024-03-25T19:41:06.496 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:06.496 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:06.496 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:41:06.496 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.495+0000 7ff27658c640 1 librados: starting msgr at 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.495+0000 7ff27658c640 1 librados: starting objecter 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.495+0000 7ff27658c640 1 Processor -- start 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 1 -- start start 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 1 librados: setting wanted keys 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 1 librados: calling monclient init 2024-03-25T19:41:06.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 10 monclient: init 2024-03-25T19:41:06.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:06.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:06.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 10 monclient(hunting): picked mon.noname-c con 0x7ff2700a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:06.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 10 monclient(hunting): picked mon.noname-b con 0x7ff27013f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:06.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 10 monclient(hunting): picked mon.noname-a con 0x7ff2701430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:06.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff270137ec0 con 0x7ff2701430c0 2024-03-25T19:41:06.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff270136dd0 con 0x7ff27013f710 2024-03-25T19:41:06.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2701370b0 con 0x7ff2700a98f0 2024-03-25T19:41:06.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:06.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.496+0000 7ff27658c640 10 monclient(hunting): authenticate will time out at 4858.837684s 2024-03-25T19:41:06.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.499+0000 7ff26ffff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff2700a98f0 0x7ff2701377b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39270/0 (socket says 172.21.15.107:39270) 2024-03-25T19:41:06.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.499+0000 7ff26ffff640 1 -- 172.21.15.107:0/1375011355 learned_addr learned my addr 172.21.15.107:0/1375011355 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:06.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.501+0000 7ff26cff9640 10 client.?.objecter ms_handle_connect 0x7ff2700a98f0 2024-03-25T19:41:06.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.501+0000 7ff26cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:06.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.514+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205506898 0 0) 0x7ff2701370b0 con 0x7ff2700a98f0 2024-03-25T19:41:06.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.514+0000 7ff26cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:06.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.514+0000 7ff26cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:06.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.514+0000 7ff26cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:06.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.514+0000 7ff26cff9640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:41:06.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.514+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff25c004eb0 con 0x7ff2700a98f0 2024-03-25T19:41:06.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.521+0000 7ff26cff9640 10 client.?.objecter ms_handle_connect 0x7ff27013f710 2024-03-25T19:41:06.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.521+0000 7ff26cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:06.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.524+0000 7ff26cff9640 10 client.?.objecter ms_handle_connect 0x7ff2701430c0 2024-03-25T19:41:06.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.524+0000 7ff26cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:06.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.529+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1548034724 0 0) 0x7ff25c004eb0 con 0x7ff2700a98f0 2024-03-25T19:41:06.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.529+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2701370b0 con 0x7ff2700a98f0 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.529+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff260001510 con 0x7ff2700a98f0 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.529+0000 7ff26cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.529+0000 7ff26cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.529+0000 7ff26cff9640 10 monclient(hunting): dump: 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:06.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.529+0000 7ff26cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:06.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.531+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024364367 0 0) 0x7ff270137ec0 con 0x7ff2701430c0 2024-03-25T19:41:06.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.531+0000 7ff26cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:06.531 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.531+0000 7ff26cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:06.531 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.531+0000 7ff26cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:06.531 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.531+0000 7ff26cff9640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:41:06.531 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.531+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff25c004eb0 con 0x7ff2701430c0 2024-03-25T19:41:06.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.536+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225627380 0 0) 0x7ff270136dd0 con 0x7ff27013f710 2024-03-25T19:41:06.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.536+0000 7ff26cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:06.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.536+0000 7ff26cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:06.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.536+0000 7ff26cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:06.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.536+0000 7ff26cff9640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:41:06.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.536+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff270137ec0 con 0x7ff27013f710 2024-03-25T19:41:06.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.548+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678545300 0 0) 0x7ff270137ec0 con 0x7ff27013f710 2024-03-25T19:41:06.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.548+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff270136dd0 con 0x7ff27013f710 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.548+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff264001b00 con 0x7ff27013f710 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.548+0000 7ff26cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.548+0000 7ff26cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.548+0000 7ff26cff9640 10 monclient(hunting): dump: 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:06.548 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:06.549 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.549+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1162744347 0 0) 0x7ff25c004eb0 con 0x7ff2701430c0 2024-03-25T19:41:06.549 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.549+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff270137ec0 con 0x7ff2701430c0 2024-03-25T19:41:06.549 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.549+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff268001b90 con 0x7ff2701430c0 2024-03-25T19:41:06.549 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.549+0000 7ff26cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:06.549 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.549+0000 7ff26cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:06.549 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.549+0000 7ff26cff9640 10 monclient(hunting): dump: 2024-03-25T19:41:06.549 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:06.549 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:06.550 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.550 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.550 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:06.550 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:06.550 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:06.550 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:06.550 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:06.550 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:06.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1239879703 0 0) 0x7ff270136dd0 con 0x7ff27013f710 2024-03-25T19:41:06.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 >> v1:172.21.15.107:6790/0 conn(0x7ff2700a98f0 legacy=0x7ff2701377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:06.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 >> v1:172.21.15.107:6789/0 conn(0x7ff2701430c0 legacy=0x7ff2701db4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:06.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff26cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:06.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff26cff9640 1 monclient: found mon.b 2024-03-25T19:41:06.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff26cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:06.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff26cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:06.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff270137390 con 0x7ff27013f710 2024-03-25T19:41:06.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff26cff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:06.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff26cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:06.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff26cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:06.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff27658c640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:41:06.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff27658c640 10 monclient: _renew_subs 2024-03-25T19:41:06.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff27658c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:06.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff27658c640 1 -- 172.21.15.107:0/1375011355 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff270135890 con 0x7ff27013f710 2024-03-25T19:41:06.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff27658c640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:06.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff27658c640 10 monclient: _renew_subs 2024-03-25T19:41:06.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff27658c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:06.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff27658c640 1 -- 172.21.15.107:0/1375011355 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff270135100 con 0x7ff27013f710 2024-03-25T19:41:06.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.557+0000 7ff27658c640 1 librados: init done 2024-03-25T19:41:06.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.558+0000 7ff27658c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:06.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.558+0000 7ff27658c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:06.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.558+0000 7ff27658c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:06.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.558+0000 7ff27658c640 1 -- 172.21.15.107:0/1375011355 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2700627b0 con 0x7ff27013f710 2024-03-25T19:41:06.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.858+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2640038a0 con 0x7ff27013f710 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.858+0000 7ff26cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.858+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff264001250 con 0x7ff27013f710 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.859+0000 7ff26cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.859+0000 7ff26cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.859+0000 7ff26cff9640 10 monclient: dump: 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:06.859 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:06.860 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:06.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.859+0000 7ff26e7fc640 4 set_mon_vals no callback set 2024-03-25T19:41:06.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.860+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7ff264019a20 con 0x7ff27013f710 2024-03-25T19:41:06.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.860+0000 7ff26ffff640 10 monclient: get_auth_request con 0x7ff25c06e010 auth_method 0 2024-03-25T19:41:06.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.861+0000 7ff26cff9640 10 client.4450.objecter ms_handle_connect 0x7ff25c06e010 2024-03-25T19:41:06.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.864+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7ff264053580 con 0x7ff27013f710 2024-03-25T19:41:06.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.864+0000 7ff26cff9640 10 client.4450.objecter ms_dispatch 0x7ff270000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:06.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.864+0000 7ff26cff9640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:06.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.864+0000 7ff26cff9640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:06.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.864+0000 7ff26cff9640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-25T19:41:06.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.865+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff26404eec0 con 0x7ff27013f710 2024-03-25T19:41:06.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.865+0000 7ff26cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:06.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:06.865+0000 7ff26cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:07.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.115+0000 7ff27658c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:07.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.115+0000 7ff27658c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:07.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.115+0000 7ff27658c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:07.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.115+0000 7ff27658c640 1 -- 172.21.15.107:0/1375011355 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff27013dbb0 con 0x7ff27013f710 2024-03-25T19:41:07.143 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.143+0000 7ff26cff9640 1 -- 172.21.15.107:0/1375011355 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7ff264050fc0 con 0x7ff27013f710 2024-03-25T19:41:07.143 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.143+0000 7ff26cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:07.143 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.143+0000 7ff26cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:07.144 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:41:07.144 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:07.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 10 librados: watch_flush enter 2024-03-25T19:41:07.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 10 librados: watch_flush exit 2024-03-25T19:41:07.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-25T19:41:07.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 10 client.4450.objecter successfully canceled tick 2024-03-25T19:41:07.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 1 -- 172.21.15.107:0/1375011355 >> v1:172.21.15.195:6800/2984408421 conn(0x7ff25c06e010 legacy=0x7ff25c0704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:07.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 10 monclient: shutdown 2024-03-25T19:41:07.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:07.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 1 -- 172.21.15.107:0/1375011355 >> v1:172.21.15.195:6789/0 conn(0x7ff27013f710 legacy=0x7ff27013b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:07.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 1 -- 172.21.15.107:0/1375011355 shutdown_connections 2024-03-25T19:41:07.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 1 -- 172.21.15.107:0/1375011355 >> 172.21.15.107:0/1375011355 conn(0x7ff2700b7ea0 msgr2=0x7ff270074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:07.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 1 -- 172.21.15.107:0/1375011355 shutdown_connections 2024-03-25T19:41:07.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 1 -- 172.21.15.107:0/1375011355 wait complete. 2024-03-25T19:41:07.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:07.146+0000 7ff27658c640 1 librados: shutdown 2024-03-25T19:41:07.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:13.967 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:14.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.085+0000 7f90e9eba640 10 monclient: get_monmap_and_config 2024-03-25T19:41:14.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.085+0000 7f90e9eba640 10 monclient: build_initial_monmap 2024-03-25T19:41:14.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.085+0000 7f90e9eba640 10 monclient: monmap: 2024-03-25T19:41:14.085 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:14.085 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:14.085 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:41:14.086184+0000 2024-03-25T19:41:14.085 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:41:14.086184+0000 2024-03-25T19:41:14.085 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:14.085 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:14.085 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:14.085 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:41:14.085 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:41:14.086 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:14.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 1 Processor -- start 2024-03-25T19:41:14.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 1 -- start start 2024-03-25T19:41:14.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 10 monclient: init 2024-03-25T19:41:14.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:14.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:14.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 10 monclient(hunting): picked mon.noname-b con 0x7f90e40a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:14.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 10 monclient(hunting): picked mon.noname-c con 0x7f90e413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:14.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 10 monclient(hunting): picked mon.noname-a con 0x7f90e41430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:14.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e4142520 con 0x7f90e41430c0 2024-03-25T19:41:14.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e40b28a0 con 0x7f90e413f710 2024-03-25T19:41:14.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e40b2a30 con 0x7f90e40a98f0 2024-03-25T19:41:14.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:14.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.086+0000 7f90e9eba640 10 monclient(hunting): authenticate will time out at 4866.427630s 2024-03-25T19:41:14.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.096+0000 7f90e2ffd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f90e413f710 0x7f90e41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:43420/0 (socket says 172.21.15.107:43420) 2024-03-25T19:41:14.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.097+0000 7f90e2ffd640 1 -- 172.21.15.107:0/2221614889 learned_addr learned my addr 172.21.15.107:0/2221614889 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:14.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.112+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059182572 0 0) 0x7f90e40b28a0 con 0x7f90e413f710 2024-03-25T19:41:14.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.112+0000 7f90e27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:14.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.112+0000 7f90e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:14.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.112+0000 7f90e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:14.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.112+0000 7f90e27fc640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:41:14.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.112+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90cc003520 con 0x7f90e413f710 2024-03-25T19:41:14.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.113+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429342973 0 0) 0x7f90e4142520 con 0x7f90e41430c0 2024-03-25T19:41:14.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.113+0000 7f90e27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:14.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.113+0000 7f90e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:14.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.113+0000 7f90e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:14.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.113+0000 7f90e27fc640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:41:14.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.113+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e40b28a0 con 0x7f90e41430c0 2024-03-25T19:41:14.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.120+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424068328 0 0) 0x7f90e40b2a30 con 0x7f90e40a98f0 2024-03-25T19:41:14.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.120+0000 7f90e27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:14.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.120+0000 7f90e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:14.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.120+0000 7f90e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:14.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.120+0000 7f90e27fc640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:41:14.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.120+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e4142520 con 0x7f90e40a98f0 2024-03-25T19:41:14.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3247651194 0 0) 0x7f90e4142520 con 0x7f90e40a98f0 2024-03-25T19:41:14.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 >> v1:172.21.15.195:6789/0 conn(0x7f90e413f710 legacy=0x7f90e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 >> v1:172.21.15.107:6789/0 conn(0x7f90e41430c0 legacy=0x7f90e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:14.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 1 monclient: found mon.noname-b 2024-03-25T19:41:14.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:14.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:41:14.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e40b44c0 con 0x7f90e40a98f0 2024-03-25T19:41:14.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:14.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:14.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f90d40015b0 con 0x7f90e40a98f0 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e27fc640 10 monclient: dump: 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:14.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e9eba640 5 monclient: authenticate success, global_id 4466 2024-03-25T19:41:14.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.132+0000 7f90e9eba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.145+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90d4001710 con 0x7f90e40a98f0 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.145+0000 7f90e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.145+0000 7f90e27fc640 1 -- 172.21.15.107:0/2221614889 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f90d4001b20 con 0x7f90e40a98f0 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.145+0000 7f90e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.145+0000 7f90e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.145+0000 7f90e27fc640 10 monclient: dump: 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:14.145 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:14.146 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:14.146 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:14.146 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:14.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.145+0000 7f90e9eba640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:14.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.145+0000 7f90e9eba640 4 set_mon_vals no callback set 2024-03-25T19:41:14.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.146+0000 7f90e9eba640 10 monclient: shutdown 2024-03-25T19:41:14.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.146+0000 7f90e9eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:14.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.146+0000 7f90e9eba640 1 -- 172.21.15.107:0/2221614889 >> v1:172.21.15.107:6790/0 conn(0x7f90e40a98f0 legacy=0x7f90e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.146+0000 7f90e9eba640 1 -- 172.21.15.107:0/2221614889 shutdown_connections 2024-03-25T19:41:14.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.146+0000 7f90e9eba640 1 -- 172.21.15.107:0/2221614889 >> 172.21.15.107:0/2221614889 conn(0x7f90e40b7ea0 msgr2=0x7f90e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:14.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.146+0000 7f90e9eba640 1 -- 172.21.15.107:0/2221614889 shutdown_connections 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.146+0000 7f90e9eba640 1 -- 172.21.15.107:0/2221614889 wait complete. 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.147+0000 7f90e9eba640 10 monclient: build_initial_monmap 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.147+0000 7f90e9eba640 10 monclient: monmap: 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.147+0000 7f90e9eba640 1 librados: starting msgr at 2024-03-25T19:41:14.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.147+0000 7f90e9eba640 1 librados: starting objecter 2024-03-25T19:41:14.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.147+0000 7f90e9eba640 1 Processor -- start 2024-03-25T19:41:14.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.147+0000 7f90e9eba640 1 -- start start 2024-03-25T19:41:14.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.147+0000 7f90e9eba640 1 librados: setting wanted keys 2024-03-25T19:41:14.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.147+0000 7f90e9eba640 1 librados: calling monclient init 2024-03-25T19:41:14.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.147+0000 7f90e9eba640 10 monclient: init 2024-03-25T19:41:14.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.148+0000 7f90e9eba640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:14.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.148+0000 7f90e9eba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:14.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.148+0000 7f90e9eba640 10 monclient(hunting): picked mon.noname-b con 0x7f90e40a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:14.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.148+0000 7f90e9eba640 10 monclient(hunting): picked mon.noname-a con 0x7f90e413f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:14.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.148+0000 7f90e9eba640 10 monclient(hunting): picked mon.noname-c con 0x7f90e41430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:14.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.148+0000 7f90e9eba640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e41d7750 con 0x7f90e413f710 2024-03-25T19:41:14.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.148+0000 7f90e9eba640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e41d7a30 con 0x7f90e40a98f0 2024-03-25T19:41:14.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.148+0000 7f90e9eba640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e40abd30 con 0x7f90e41430c0 2024-03-25T19:41:14.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.148+0000 7f90e9eba640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:14.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.148+0000 7f90e9eba640 10 monclient(hunting): authenticate will time out at 4866.489553s 2024-03-25T19:41:14.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.156+0000 7f90e3fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f90e41430c0 0x7f90e4138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39272/0 (socket says 172.21.15.107:39272) 2024-03-25T19:41:14.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.156+0000 7f90e3fff640 1 -- 172.21.15.107:0/1469886926 learned_addr learned my addr 172.21.15.107:0/1469886926 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:14.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.159+0000 7f90cbfff640 10 client.?.objecter ms_handle_connect 0x7f90e41430c0 2024-03-25T19:41:14.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.159+0000 7f90cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:14.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.164+0000 7f90cbfff640 10 client.?.objecter ms_handle_connect 0x7f90e413f710 2024-03-25T19:41:14.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.164+0000 7f90cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:14.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.170+0000 7f90cbfff640 10 client.?.objecter ms_handle_connect 0x7f90e40a98f0 2024-03-25T19:41:14.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.170+0000 7f90cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.175+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576250598 0 0) 0x7f90e41d7750 con 0x7f90e413f710 2024-03-25T19:41:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.175+0000 7f90cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.175+0000 7f90cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.175+0000 7f90cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:14.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.175+0000 7f90cbfff640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:41:14.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.175+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d0004eb0 con 0x7f90e413f710 2024-03-25T19:41:14.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.180+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299832521 0 0) 0x7f90e40abd30 con 0x7f90e41430c0 2024-03-25T19:41:14.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.180+0000 7f90cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:14.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.180+0000 7f90cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:14.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.180+0000 7f90cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:14.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.180+0000 7f90cbfff640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:41:14.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.180+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e41d7750 con 0x7f90e41430c0 2024-03-25T19:41:14.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.189+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596858661 0 0) 0x7f90e41d7a30 con 0x7f90e40a98f0 2024-03-25T19:41:14.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.189+0000 7f90cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:14.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.189+0000 7f90cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:14.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.189+0000 7f90cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:14.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.189+0000 7f90cbfff640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:41:14.189 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.189+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e40abd30 con 0x7f90e40a98f0 2024-03-25T19:41:14.200 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3270753374 0 0) 0x7f90d0004eb0 con 0x7f90e413f710 2024-03-25T19:41:14.200 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90e41d7a30 con 0x7f90e413f710 2024-03-25T19:41:14.200 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f90d8001950 con 0x7f90e413f710 2024-03-25T19:41:14.200 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:14.200 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:14.200 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 10 monclient(hunting): dump: 2024-03-25T19:41:14.200 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:14.200 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:14.200 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091448322 0 0) 0x7f90e41d7750 con 0x7f90e41430c0 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90d0004eb0 con 0x7f90e41430c0 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f90dc001c50 con 0x7f90e41430c0 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.200+0000 7f90cbfff640 10 monclient(hunting): dump: 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:14.201 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:14.202 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.202 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.202 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:14.202 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:14.202 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:14.202 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:14.202 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:14.202 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:14.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.210+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2853852233 0 0) 0x7f90e40abd30 con 0x7f90e40a98f0 2024-03-25T19:41:14.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.211+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90e41d7750 con 0x7f90e40a98f0 2024-03-25T19:41:14.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.211+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f90d4001a20 con 0x7f90e40a98f0 2024-03-25T19:41:14.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.211+0000 7f90cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.211+0000 7f90cbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.211+0000 7f90cbfff640 10 monclient(hunting): dump: 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:14.211 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:14.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2901849484 0 0) 0x7f90d0004eb0 con 0x7f90e41430c0 2024-03-25T19:41:14.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 >> v1:172.21.15.195:6789/0 conn(0x7f90e40a98f0 legacy=0x7f90e41d7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 >> v1:172.21.15.107:6789/0 conn(0x7f90e413f710 legacy=0x7f90e41d4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90cbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90cbfff640 1 monclient: found mon.c 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90cbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e41d85d0 con 0x7f90e41430c0 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90cbfff640 10 monclient: _finish_auth 0 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90cbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90e9eba640 5 monclient: authenticate success, global_id 4469 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90e9eba640 10 monclient: _renew_subs 2024-03-25T19:41:14.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90e9eba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:41:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.216+0000 7f90e9eba640 1 -- 172.21.15.107:0/1469886926 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e41d51a0 con 0x7f90e41430c0 2024-03-25T19:41:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.217+0000 7f90e9eba640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.217+0000 7f90e9eba640 10 monclient: _renew_subs 2024-03-25T19:41:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.217+0000 7f90e9eba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:41:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.217+0000 7f90e9eba640 1 -- 172.21.15.107:0/1469886926 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e41d43c0 con 0x7f90e41430c0 2024-03-25T19:41:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.217+0000 7f90e9eba640 1 librados: init done 2024-03-25T19:41:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.218+0000 7f90e9eba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:14.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.218+0000 7f90e9eba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:14.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.218+0000 7f90e9eba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:41:14.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.218+0000 7f90e9eba640 1 -- 172.21.15.107:0/1469886926 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e40627b0 con 0x7f90e41430c0 2024-03-25T19:41:14.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.232+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90dc003970 con 0x7f90e41430c0 2024-03-25T19:41:14.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.232+0000 7f90cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:14.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.232+0000 7f90e1ffb640 4 set_mon_vals no callback set 2024-03-25T19:41:14.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.513+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f90dc003cc0 con 0x7f90e41430c0 2024-03-25T19:41:14.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.513+0000 7f90cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:14.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.513+0000 7f90cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:14.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.513+0000 7f90cbfff640 10 monclient: dump: 2024-03-25T19:41:14.513 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:14.513 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:14.513 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.513 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:14.514 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:14.514 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:14.514 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:14.514 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:14.514 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:14.514 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:14.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.514+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f90dc003ea0 con 0x7f90e41430c0 2024-03-25T19:41:14.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.515+0000 7f90e37fe640 10 monclient: get_auth_request con 0x7f90d006e110 auth_method 0 2024-03-25T19:41:14.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.515+0000 7f90cbfff640 10 client.4469.objecter ms_handle_connect 0x7f90d006e110 2024-03-25T19:41:14.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.521+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f90dc054a70 con 0x7f90e41430c0 2024-03-25T19:41:14.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.521+0000 7f90cbfff640 10 client.4469.objecter ms_dispatch 0x7f90e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:14.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.521+0000 7f90cbfff640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:14.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.521+0000 7f90cbfff640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:14.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.521+0000 7f90cbfff640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-25T19:41:14.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.521+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90dc050580 con 0x7f90e41430c0 2024-03-25T19:41:14.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.521+0000 7f90cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:14.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.521+0000 7f90cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:14.790 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.791+0000 7f90e9eba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:14.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.791+0000 7f90e9eba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:14.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.791+0000 7f90e9eba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:41:14.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.791+0000 7f90e9eba640 1 -- 172.21.15.107:0/1469886926 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90e4141bb0 con 0x7f90e41430c0 2024-03-25T19:41:14.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.831+0000 7f90cbfff640 1 -- 172.21.15.107:0/1469886926 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f90dc0524b0 con 0x7f90e41430c0 2024-03-25T19:41:14.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.831+0000 7f90cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:14.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.831+0000 7f90cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:14.832 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:41:14.832 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:14.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.833+0000 7f90e9eba640 10 librados: watch_flush enter 2024-03-25T19:41:14.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.833+0000 7f90e9eba640 10 librados: watch_flush exit 2024-03-25T19:41:14.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.833+0000 7f90e9eba640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-25T19:41:14.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.833+0000 7f90e9eba640 10 client.4469.objecter successfully canceled tick 2024-03-25T19:41:14.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.833+0000 7f90e9eba640 1 -- 172.21.15.107:0/1469886926 >> v1:172.21.15.195:6800/2984408421 conn(0x7f90d006e110 legacy=0x7f90d00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.834+0000 7f90e9eba640 10 monclient: shutdown 2024-03-25T19:41:14.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.834+0000 7f90e9eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:14.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.834+0000 7f90e9eba640 1 -- 172.21.15.107:0/1469886926 >> v1:172.21.15.107:6790/0 conn(0x7f90e41430c0 legacy=0x7f90e4138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:14.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.834+0000 7f90e9eba640 1 -- 172.21.15.107:0/1469886926 shutdown_connections 2024-03-25T19:41:14.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.834+0000 7f90e9eba640 1 -- 172.21.15.107:0/1469886926 >> 172.21.15.107:0/1469886926 conn(0x7f90e40b7ea0 msgr2=0x7f90e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:14.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.834+0000 7f90e9eba640 1 -- 172.21.15.107:0/1469886926 shutdown_connections 2024-03-25T19:41:14.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.834+0000 7f90e9eba640 1 -- 172.21.15.107:0/1469886926 wait complete. 2024-03-25T19:41:14.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:14.834+0000 7f90e9eba640 1 librados: shutdown 2024-03-25T19:41:14.845 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:21.651 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:21.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.761+0000 7ff7881a4640 10 monclient: get_monmap_and_config 2024-03-25T19:41:21.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.761+0000 7ff7881a4640 10 monclient: build_initial_monmap 2024-03-25T19:41:21.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.761+0000 7ff7881a4640 10 monclient: monmap: 2024-03-25T19:41:21.762 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:21.762 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:21.762 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:41:21.762803+0000 2024-03-25T19:41:21.762 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:41:21.762803+0000 2024-03-25T19:41:21.762 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:21.762 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:21.762 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:21.762 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:41:21.762 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:41:21.762 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:21.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.762+0000 7ff7881a4640 1 Processor -- start 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.762+0000 7ff7881a4640 1 -- start start 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.762+0000 7ff7881a4640 10 monclient: init 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.763+0000 7ff7881a4640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.763+0000 7ff7881a4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.763+0000 7ff7881a4640 10 monclient(hunting): picked mon.noname-a con 0x7ff7800a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.763+0000 7ff7881a4640 10 monclient(hunting): picked mon.noname-c con 0x7ff78013f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.763+0000 7ff7881a4640 10 monclient(hunting): picked mon.noname-b con 0x7ff7801430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.763+0000 7ff7881a4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff780142520 con 0x7ff7800a98f0 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.763+0000 7ff7881a4640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7800b28a0 con 0x7ff78013f710 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.763+0000 7ff7881a4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7800b2a30 con 0x7ff7801430c0 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.763+0000 7ff7881a4640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:21.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.763+0000 7ff7881a4640 10 monclient(hunting): authenticate will time out at 4874.104281s 2024-03-25T19:41:21.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.778+0000 7ff78573f640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff78013f710 0x7ff7801429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:49822/0 (socket says 172.21.15.107:49822) 2024-03-25T19:41:21.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.778+0000 7ff78573f640 1 -- 172.21.15.107:0/1094952075 learned_addr learned my addr 172.21.15.107:0/1094952075 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:21.792 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.793+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147223194 0 0) 0x7ff7800b28a0 con 0x7ff78013f710 2024-03-25T19:41:21.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.793+0000 7ff784f3e640 10 monclient(hunting): _negotiate 2024-03-25T19:41:21.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.793+0000 7ff784f3e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:21.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.793+0000 7ff784f3e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:21.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.793+0000 7ff784f3e640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:41:21.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.793+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff76c003520 con 0x7ff78013f710 2024-03-25T19:41:21.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775806043 0 0) 0x7ff7800b2a30 con 0x7ff7801430c0 2024-03-25T19:41:21.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 10 monclient(hunting): _negotiate 2024-03-25T19:41:21.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:21.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:21.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:41:21.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7800b28a0 con 0x7ff7801430c0 2024-03-25T19:41:21.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211358564 0 0) 0x7ff780142520 con 0x7ff7800a98f0 2024-03-25T19:41:21.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 10 monclient(hunting): _negotiate 2024-03-25T19:41:21.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:21.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:21.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:41:21.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.799+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7800b2a30 con 0x7ff7800a98f0 2024-03-25T19:41:21.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2581100121 0 0) 0x7ff76c003520 con 0x7ff78013f710 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 >> v1:172.21.15.107:6790/0 conn(0x7ff7801430c0 legacy=0x7ff78014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 >> v1:172.21.15.107:6789/0 conn(0x7ff7800a98f0 legacy=0x7ff7800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 10 monclient: _finish_hunting 0 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 1 monclient: found mon.noname-c 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7800b44c0 con 0x7ff78013f710 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 10 monclient: _finish_auth 0 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 10 monclient: _check_auth_tickets 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff770001690 con 0x7ff78013f710 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff784f3e640 10 monclient: dump: 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:21.801 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:21.802 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:21.802 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:21.802 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:21.802 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:21.802 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:21.802 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:21.802 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:21.802 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:21.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff7881a4640 5 monclient: authenticate success, global_id 4459 2024-03-25T19:41:21.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.800+0000 7ff7881a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:21.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.810+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7700017f0 con 0x7ff78013f710 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.810+0000 7ff784f3e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.810+0000 7ff784f3e640 1 -- 172.21.15.107:0/1094952075 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff770001c00 con 0x7ff78013f710 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.810+0000 7ff7881a4640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.810+0000 7ff7881a4640 4 set_mon_vals no callback set 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.810+0000 7ff784f3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.810+0000 7ff784f3e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.810+0000 7ff784f3e640 10 monclient: dump: 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:21.810 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:21.811 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:21.811 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:21.811 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:21.811 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:21.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.811+0000 7ff7881a4640 10 monclient: shutdown 2024-03-25T19:41:21.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.811+0000 7ff7881a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.811+0000 7ff7881a4640 1 -- 172.21.15.107:0/1094952075 >> v1:172.21.15.195:6789/0 conn(0x7ff78013f710 legacy=0x7ff7801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.811+0000 7ff7881a4640 1 -- 172.21.15.107:0/1094952075 shutdown_connections 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.811+0000 7ff7881a4640 1 -- 172.21.15.107:0/1094952075 >> 172.21.15.107:0/1094952075 conn(0x7ff7800b7ea0 msgr2=0x7ff7800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.811+0000 7ff7881a4640 1 -- 172.21.15.107:0/1094952075 shutdown_connections 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.811+0000 7ff7881a4640 1 -- 172.21.15.107:0/1094952075 wait complete. 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.812+0000 7ff7881a4640 10 monclient: build_initial_monmap 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.812+0000 7ff7881a4640 10 monclient: monmap: 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:21.812 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:41:21.813 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:21.813 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:41:21.813 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:41:21.813 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:21.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.812+0000 7ff7881a4640 1 librados: starting msgr at 2024-03-25T19:41:21.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.812+0000 7ff7881a4640 1 librados: starting objecter 2024-03-25T19:41:21.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.812+0000 7ff7881a4640 1 Processor -- start 2024-03-25T19:41:21.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 1 -- start start 2024-03-25T19:41:21.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 1 librados: setting wanted keys 2024-03-25T19:41:21.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 1 librados: calling monclient init 2024-03-25T19:41:21.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 10 monclient: init 2024-03-25T19:41:21.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:21.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:21.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 10 monclient(hunting): picked mon.noname-c con 0x7ff7800a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:21.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 10 monclient(hunting): picked mon.noname-a con 0x7ff78013f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:21.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 10 monclient(hunting): picked mon.noname-b con 0x7ff7801430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:21.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff780138a50 con 0x7ff78013f710 2024-03-25T19:41:21.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7800ab9e0 con 0x7ff7801430c0 2024-03-25T19:41:21.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7801d7880 con 0x7ff7800a98f0 2024-03-25T19:41:21.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:21.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.813+0000 7ff7881a4640 10 monclient(hunting): authenticate will time out at 4874.154708s 2024-03-25T19:41:21.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.817+0000 7ff786741640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff7801430c0 0x7ff7801e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:49854/0 (socket says 172.21.15.107:49854) 2024-03-25T19:41:21.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.817+0000 7ff786741640 1 -- 172.21.15.107:0/1279097096 learned_addr learned my addr 172.21.15.107:0/1279097096 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:21.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.818+0000 7ff7767fc640 10 client.?.objecter ms_handle_connect 0x7ff7801430c0 2024-03-25T19:41:21.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.818+0000 7ff7767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:21.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.821+0000 7ff7767fc640 10 client.?.objecter ms_handle_connect 0x7ff7800a98f0 2024-03-25T19:41:21.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.821+0000 7ff7767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:21.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.830+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358596681 0 0) 0x7ff7801d7880 con 0x7ff7800a98f0 2024-03-25T19:41:21.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.830+0000 7ff7767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:21.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.830+0000 7ff7767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:21.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.830+0000 7ff7767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:21.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.830+0000 7ff7767fc640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:41:21.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.830+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff764004eb0 con 0x7ff7800a98f0 2024-03-25T19:41:21.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.831+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484174567 0 0) 0x7ff7800ab9e0 con 0x7ff7801430c0 2024-03-25T19:41:21.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.831+0000 7ff7767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:21.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.831+0000 7ff7767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:21.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.831+0000 7ff7767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:21.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.831+0000 7ff7767fc640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:41:21.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.831+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7801d7880 con 0x7ff7801430c0 2024-03-25T19:41:21.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.834+0000 7ff7767fc640 10 client.?.objecter ms_handle_connect 0x7ff78013f710 2024-03-25T19:41:21.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.834+0000 7ff7767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.843+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3087853566 0 0) 0x7ff764004eb0 con 0x7ff7800a98f0 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.843+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7800ab9e0 con 0x7ff7800a98f0 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.843+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff77c0018f0 con 0x7ff7800a98f0 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.843+0000 7ff7767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.843+0000 7ff7767fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.843+0000 7ff7767fc640 10 monclient(hunting): dump: 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:21.843 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:21.844 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:21.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.843+0000 7ff7767fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:21.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.845+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309390068 0 0) 0x7ff780138a50 con 0x7ff78013f710 2024-03-25T19:41:21.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.845+0000 7ff7767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:21.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.845+0000 7ff7767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:21.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.845+0000 7ff7767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:21.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.845+0000 7ff7767fc640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:41:21.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.845+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff764004eb0 con 0x7ff78013f710 2024-03-25T19:41:21.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.849+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2184992130 0 0) 0x7ff7801d7880 con 0x7ff7801430c0 2024-03-25T19:41:21.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.849+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff780138a50 con 0x7ff7801430c0 2024-03-25T19:41:21.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.849+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff778001c50 con 0x7ff7801430c0 2024-03-25T19:41:21.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.849+0000 7ff7767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:21.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.849+0000 7ff7767fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:21.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.849+0000 7ff7767fc640 10 monclient(hunting): dump: 2024-03-25T19:41:21.849 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:21.849 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:21.849 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:21.849 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:21.849 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:21.850 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:21.850 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:21.850 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:21.850 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:21.850 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:21.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 252679678 0 0) 0x7ff7800ab9e0 con 0x7ff7800a98f0 2024-03-25T19:41:21.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 >> v1:172.21.15.195:6789/0 conn(0x7ff7801430c0 legacy=0x7ff7801e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:21.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 >> v1:172.21.15.107:6789/0 conn(0x7ff78013f710 legacy=0x7ff7801d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7767fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7767fc640 1 monclient: found mon.c 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7767fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7801d84b0 con 0x7ff7800a98f0 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7767fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7767fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7881a4640 5 monclient: authenticate success, global_id 4475 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7881a4640 10 monclient: _renew_subs 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7881a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:41:21.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7881a4640 1 -- 172.21.15.107:0/1279097096 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7801d51f0 con 0x7ff7800a98f0 2024-03-25T19:41:21.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7881a4640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:21.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7881a4640 10 monclient: _renew_subs 2024-03-25T19:41:21.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7881a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:41:21.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7881a4640 1 -- 172.21.15.107:0/1279097096 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff780138e60 con 0x7ff7800a98f0 2024-03-25T19:41:21.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.853+0000 7ff7881a4640 1 librados: init done 2024-03-25T19:41:21.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.855+0000 7ff7881a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:21.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.855+0000 7ff7881a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:21.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.855+0000 7ff7881a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:41:21.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:21.855+0000 7ff7881a4640 1 -- 172.21.15.107:0/1279097096 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7801d76e0 con 0x7ff7800a98f0 2024-03-25T19:41:22.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.157+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff77c003610 con 0x7ff7800a98f0 2024-03-25T19:41:22.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.157+0000 7ff7767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:22.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.157+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff77c003780 con 0x7ff7800a98f0 2024-03-25T19:41:22.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.157+0000 7ff7767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:22.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.157+0000 7ff7767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:22.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.157+0000 7ff7767fc640 10 monclient: dump: 2024-03-25T19:41:22.157 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:22.158 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:22.158 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:22.158 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:22.158 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:22.158 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:22.158 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:22.158 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:22.158 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:22.158 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:22.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.157+0000 7ff777fff640 4 set_mon_vals no callback set 2024-03-25T19:41:22.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.158+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7ff77c019790 con 0x7ff7800a98f0 2024-03-25T19:41:22.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.159+0000 7ff78573f640 10 monclient: get_auth_request con 0x7ff76406dfb0 auth_method 0 2024-03-25T19:41:22.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.159+0000 7ff7767fc640 10 client.4475.objecter ms_handle_connect 0x7ff76406dfb0 2024-03-25T19:41:22.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.164+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7ff77c053450 con 0x7ff7800a98f0 2024-03-25T19:41:22.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.164+0000 7ff7767fc640 10 client.4475.objecter ms_dispatch 0x7ff780000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:22.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.164+0000 7ff7767fc640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:22.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.164+0000 7ff7767fc640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:22.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.164+0000 7ff7767fc640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-25T19:41:22.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.164+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff77c04ed90 con 0x7ff7800a98f0 2024-03-25T19:41:22.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.164+0000 7ff7767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:22.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.164+0000 7ff7767fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:22.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.398+0000 7ff7881a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:22.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.398+0000 7ff7881a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:22.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.398+0000 7ff7881a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:41:22.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.398+0000 7ff7881a4640 1 -- 172.21.15.107:0/1279097096 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7801d7a10 con 0x7ff7800a98f0 2024-03-25T19:41:22.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.432+0000 7ff7767fc640 1 -- 172.21.15.107:0/1279097096 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7ff77c050e90 con 0x7ff7800a98f0 2024-03-25T19:41:22.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.432+0000 7ff7767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:22.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.432+0000 7ff7767fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:22.433 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:41:22.433 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:22.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 10 librados: watch_flush enter 2024-03-25T19:41:22.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 10 librados: watch_flush exit 2024-03-25T19:41:22.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-25T19:41:22.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 10 client.4475.objecter successfully canceled tick 2024-03-25T19:41:22.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 1 -- 172.21.15.107:0/1279097096 >> v1:172.21.15.195:6800/2984408421 conn(0x7ff76406dfb0 legacy=0x7ff764070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:22.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 10 monclient: shutdown 2024-03-25T19:41:22.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:22.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 1 -- 172.21.15.107:0/1279097096 >> v1:172.21.15.107:6790/0 conn(0x7ff7800a98f0 legacy=0x7ff780138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:22.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 1 -- 172.21.15.107:0/1279097096 shutdown_connections 2024-03-25T19:41:22.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 1 -- 172.21.15.107:0/1279097096 >> 172.21.15.107:0/1279097096 conn(0x7ff7800b7ea0 msgr2=0x7ff780074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:22.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 1 -- 172.21.15.107:0/1279097096 shutdown_connections 2024-03-25T19:41:22.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 1 -- 172.21.15.107:0/1279097096 wait complete. 2024-03-25T19:41:22.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:22.435+0000 7ff7881a4640 1 librados: shutdown 2024-03-25T19:41:22.447 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:25.206 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:05:55.522 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.258 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:05:55.574 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.305 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:05:55.621 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.306 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:05:55.593 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.309 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:05:55.626 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.311 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:05:55.598 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.321 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:05:55.638 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.335 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:05:55.622 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.349 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:05:55.665 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:29.254 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.366+0000 7f51d8d4d640 10 monclient: get_monmap_and_config 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.366+0000 7f51d8d4d640 10 monclient: build_initial_monmap 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.366+0000 7f51d8d4d640 10 monclient: monmap: 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:41:29.367511+0000 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:41:29.367511+0000 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:41:29.366 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:41:29.367 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:29.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.367+0000 7f51d8d4d640 1 Processor -- start 2024-03-25T19:41:29.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.367+0000 7f51d8d4d640 1 -- start start 2024-03-25T19:41:29.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.367+0000 7f51d8d4d640 10 monclient: init 2024-03-25T19:41:29.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.368+0000 7f51d8d4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:29.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.368+0000 7f51d8d4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:29.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.368+0000 7f51d8d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f51d40b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:29.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.368+0000 7f51d8d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f51d40b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:29.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.368+0000 7f51d8d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f51d414be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:29.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.368+0000 7f51d8d4d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40af940 con 0x7f51d40b4d60 2024-03-25T19:41:29.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.368+0000 7f51d8d4d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40b0a70 con 0x7f51d40b1730 2024-03-25T19:41:29.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.368+0000 7f51d8d4d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40b0d50 con 0x7f51d414be90 2024-03-25T19:41:29.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.368+0000 7f51d8d4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:29.368 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.368+0000 7f51d8d4d640 10 monclient(hunting): authenticate will time out at 4881.708925s 2024-03-25T19:41:29.374 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.374+0000 7f51d259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f51d40b4d60 0x7f51d413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40318/0 (socket says 172.21.15.107:40318) 2024-03-25T19:41:29.374 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.374+0000 7f51d259c640 1 -- 172.21.15.107:0/2626269228 learned_addr learned my addr 172.21.15.107:0/2626269228 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:29.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.399+0000 7f51d159a640 1 -- 172.21.15.107:0/2626269228 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762803708 0 0) 0x7f51d40b0d50 con 0x7f51d414be90 2024-03-25T19:41:29.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.399+0000 7f51d159a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:29.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.399+0000 7f51d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:29.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.399+0000 7f51d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:29.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.399+0000 7f51d159a640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:41:29.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.399+0000 7f51d159a640 1 -- 172.21.15.107:0/2626269228 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51bc003520 con 0x7f51d414be90 2024-03-25T19:41:29.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.463+0000 7f51d159a640 1 -- 172.21.15.107:0/2626269228 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475543652 0 0) 0x7f51d40af940 con 0x7f51d40b4d60 2024-03-25T19:41:29.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.464+0000 7f51d159a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:29.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.464+0000 7f51d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:29.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.464+0000 7f51d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:29.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.464+0000 7f51d159a640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:41:29.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.464+0000 7f51d159a640 1 -- 172.21.15.107:0/2626269228 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d40b0d50 con 0x7f51d40b4d60 2024-03-25T19:41:29.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 1 -- 172.21.15.107:0/2626269228 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3413302072 0 0) 0x7f51bc003520 con 0x7f51d414be90 2024-03-25T19:41:29.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 1 -- 172.21.15.107:0/2626269228 >> v1:172.21.15.195:6789/0 conn(0x7f51d40b1730 legacy=0x7f51d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:29.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 1 -- 172.21.15.107:0/2626269228 >> v1:172.21.15.107:6789/0 conn(0x7f51d40b4d60 legacy=0x7f51d413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 10 monclient: _finish_hunting 0 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 1 monclient: found mon.noname-b 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 1 -- 172.21.15.107:0/2626269228 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40b1030 con 0x7f51d414be90 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 10 monclient: _finish_auth 0 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 10 monclient: _check_auth_tickets 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 1 -- 172.21.15.107:0/2626269228 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f51cc001a10 con 0x7f51d414be90 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d8d4d640 5 monclient: authenticate success, global_id 4478 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d8d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:29.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.473+0000 7f51d159a640 10 monclient: dump: 2024-03-25T19:41:29.475 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:29.475 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:29.475 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:29.475 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:29.475 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:29.475 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:29.475 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:29.475 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:29.475 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:29.475 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:29.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.474+0000 7f51d8d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:29.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.485+0000 7f51d159a640 1 -- 172.21.15.107:0/2626269228 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51cc001b70 con 0x7f51d414be90 2024-03-25T19:41:29.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.485+0000 7f51d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:29.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.485+0000 7f51d159a640 1 -- 172.21.15.107:0/2626269228 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f51cc001f80 con 0x7f51d414be90 2024-03-25T19:41:29.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.485+0000 7f51d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:29.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.485+0000 7f51d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:29.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.485+0000 7f51d159a640 10 monclient: dump: 2024-03-25T19:41:29.485 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:29.486 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:29.486 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:29.486 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:29.486 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:29.486 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:29.486 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:29.486 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:29.486 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:29.486 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:29.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.485+0000 7f51d8d4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:29.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.485+0000 7f51d8d4d640 4 set_mon_vals no callback set 2024-03-25T19:41:29.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.486+0000 7f51d8d4d640 10 monclient: shutdown 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.486+0000 7f51d8d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.486+0000 7f51d8d4d640 1 -- 172.21.15.107:0/2626269228 >> v1:172.21.15.107:6790/0 conn(0x7f51d414be90 legacy=0x7f51d414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.486+0000 7f51d8d4d640 1 -- 172.21.15.107:0/2626269228 shutdown_connections 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.486+0000 7f51d8d4d640 1 -- 172.21.15.107:0/2626269228 >> 172.21.15.107:0/2626269228 conn(0x7f51d40617a0 msgr2=0x7f51d4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.486+0000 7f51d8d4d640 1 -- 172.21.15.107:0/2626269228 shutdown_connections 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.486+0000 7f51d8d4d640 1 -- 172.21.15.107:0/2626269228 wait complete. 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.487+0000 7f51d8d4d640 10 monclient: build_initial_monmap 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.487+0000 7f51d8d4d640 10 monclient: monmap: 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:41:29.488 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:29.489 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:41:29.489 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:41:29.489 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:29.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.488+0000 7f51d8d4d640 1 librados: starting msgr at 2024-03-25T19:41:29.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.488+0000 7f51d8d4d640 1 librados: starting objecter 2024-03-25T19:41:29.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.488+0000 7f51d8d4d640 1 Processor -- start 2024-03-25T19:41:29.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.488+0000 7f51d8d4d640 1 -- start start 2024-03-25T19:41:29.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.488+0000 7f51d8d4d640 1 librados: setting wanted keys 2024-03-25T19:41:29.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.488+0000 7f51d8d4d640 1 librados: calling monclient init 2024-03-25T19:41:29.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.488+0000 7f51d8d4d640 10 monclient: init 2024-03-25T19:41:29.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.489+0000 7f51d8d4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:29.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.489+0000 7f51d8d4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:29.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.489+0000 7f51d8d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f51d40b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:29.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.489+0000 7f51d8d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f51d40b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:29.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.489+0000 7f51d8d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f51d414be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:29.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.489+0000 7f51d8d4d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d4138490 con 0x7f51d414be90 2024-03-25T19:41:29.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.489+0000 7f51d8d4d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d4136cc0 con 0x7f51d40b1730 2024-03-25T19:41:29.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.489+0000 7f51d8d4d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d4140520 con 0x7f51d40b4d60 2024-03-25T19:41:29.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.489+0000 7f51d8d4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:29.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.489+0000 7f51d8d4d640 10 monclient(hunting): authenticate will time out at 4881.830345s 2024-03-25T19:41:29.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.500+0000 7f51d2d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f51d414be90 0x7f51d41e2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40354/0 (socket says 172.21.15.107:40354) 2024-03-25T19:41:29.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.500+0000 7f51d2d9d640 1 -- 172.21.15.107:0/1321489867 learned_addr learned my addr 172.21.15.107:0/1321489867 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:29.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.501+0000 7f51c2ffd640 10 client.?.objecter ms_handle_connect 0x7f51d414be90 2024-03-25T19:41:29.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.501+0000 7f51c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:29.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.501+0000 7f51c2ffd640 10 client.?.objecter ms_handle_connect 0x7f51d40b4d60 2024-03-25T19:41:29.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.501+0000 7f51c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:29.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.504+0000 7f51c2ffd640 10 client.?.objecter ms_handle_connect 0x7f51d40b1730 2024-03-25T19:41:29.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.504+0000 7f51c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:29.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.511+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143576332 0 0) 0x7f51d4138490 con 0x7f51d414be90 2024-03-25T19:41:29.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.511+0000 7f51c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:29.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.511+0000 7f51c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:29.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.511+0000 7f51c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:29.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.511+0000 7f51c2ffd640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:41:29.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.511+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b4004eb0 con 0x7f51d414be90 2024-03-25T19:41:29.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.515+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654926677 0 0) 0x7f51d4140520 con 0x7f51d40b4d60 2024-03-25T19:41:29.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.515+0000 7f51c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:29.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.515+0000 7f51c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:29.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.515+0000 7f51c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:29.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.515+0000 7f51c2ffd640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:41:29.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.515+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d4138490 con 0x7f51d40b4d60 2024-03-25T19:41:29.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.516+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092672254 0 0) 0x7f51d4136cc0 con 0x7f51d40b1730 2024-03-25T19:41:29.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.516+0000 7f51c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:29.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.516+0000 7f51c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:29.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.516+0000 7f51c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:29.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.516+0000 7f51c2ffd640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:41:29.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.516+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d4140520 con 0x7f51d40b1730 2024-03-25T19:41:29.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.523+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1224376059 0 0) 0x7f51b4004eb0 con 0x7f51d414be90 2024-03-25T19:41:29.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.523+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51d4136cc0 con 0x7f51d414be90 2024-03-25T19:41:29.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.523+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f51cc001f80 con 0x7f51d414be90 2024-03-25T19:41:29.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.523+0000 7f51c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:29.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.523+0000 7f51c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:29.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.523+0000 7f51c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:29.523 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:29.524 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:29.524 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:29.524 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:29.524 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:29.524 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:29.524 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:29.524 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:29.524 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:29.524 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:29.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.523+0000 7f51c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:29.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.525+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 875286002 0 0) 0x7f51d4138490 con 0x7f51d40b4d60 2024-03-25T19:41:29.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.525+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51b4004eb0 con 0x7f51d40b4d60 2024-03-25T19:41:29.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.525+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f51c80018a0 con 0x7f51d40b4d60 2024-03-25T19:41:29.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.525+0000 7f51c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:29.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.525+0000 7f51c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:29.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.525+0000 7f51c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:29.526 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:29.526 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:29.526 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:29.526 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:29.526 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:29.526 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:29.526 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:29.526 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:29.526 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:29.526 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2109657236 0 0) 0x7f51d4136cc0 con 0x7f51d414be90 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 >> v1:172.21.15.107:6790/0 conn(0x7f51d40b4d60 legacy=0x7f51d4144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 >> v1:172.21.15.195:6789/0 conn(0x7f51d40b1730 legacy=0x7f51d4140940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51c2ffd640 1 monclient: found mon.a 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51c2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d4138700 con 0x7f51d414be90 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51c2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51d8d4d640 5 monclient: authenticate success, global_id 4479 2024-03-25T19:41:29.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51d8d4d640 10 monclient: _renew_subs 2024-03-25T19:41:29.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51d8d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:41:29.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51d8d4d640 1 -- 172.21.15.107:0/1321489867 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d4136660 con 0x7f51d414be90 2024-03-25T19:41:29.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51d8d4d640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:29.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51d8d4d640 10 monclient: _renew_subs 2024-03-25T19:41:29.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51d8d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:41:29.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51d8d4d640 1 -- 172.21.15.107:0/1321489867 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d4137530 con 0x7f51d414be90 2024-03-25T19:41:29.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.536+0000 7f51d8d4d640 1 librados: init done 2024-03-25T19:41:29.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.537+0000 7f51d8d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:29.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.537+0000 7f51d8d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:29.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.537+0000 7f51d8d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:41:29.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.537+0000 7f51d8d4d640 1 -- 172.21.15.107:0/1321489867 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d4136ec0 con 0x7f51d414be90 2024-03-25T19:41:29.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.830+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51cc003c20 con 0x7f51d414be90 2024-03-25T19:41:29.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.830+0000 7f51c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:29.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.830+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f51cc003dd0 con 0x7f51d414be90 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.830+0000 7f51c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.830+0000 7f51c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.830+0000 7f51c2ffd640 10 monclient: dump: 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:29.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.830+0000 7f51d0d99640 4 set_mon_vals no callback set 2024-03-25T19:41:29.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.831+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f51cc003ff0 con 0x7f51d414be90 2024-03-25T19:41:29.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.832+0000 7f51d259c640 10 monclient: get_auth_request con 0x7f51b406df70 auth_method 0 2024-03-25T19:41:29.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.832+0000 7f51c2ffd640 10 client.4479.objecter ms_handle_connect 0x7f51b406df70 2024-03-25T19:41:29.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.837+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f51cc054980 con 0x7f51d414be90 2024-03-25T19:41:29.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.837+0000 7f51c2ffd640 10 client.4479.objecter ms_dispatch 0x7f51d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:29.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.837+0000 7f51c2ffd640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:29.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.837+0000 7f51c2ffd640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:29.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.837+0000 7f51c2ffd640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-25T19:41:29.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.837+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51cc050490 con 0x7f51d414be90 2024-03-25T19:41:29.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.837+0000 7f51c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:29.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:29.837+0000 7f51c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:30.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.088+0000 7f51d8d4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:30.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.088+0000 7f51d8d4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:30.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.088+0000 7f51d8d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:41:30.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.088+0000 7f51d8d4d640 1 -- 172.21.15.107:0/1321489867 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51d41369b0 con 0x7f51d414be90 2024-03-25T19:41:30.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.124+0000 7f51c2ffd640 1 -- 172.21.15.107:0/1321489867 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f51cc0523c0 con 0x7f51d414be90 2024-03-25T19:41:30.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.124+0000 7f51c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:30.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.124+0000 7f51c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:30.124 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:41:30.124 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:30.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.126+0000 7f51d8d4d640 10 librados: watch_flush enter 2024-03-25T19:41:30.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.126+0000 7f51d8d4d640 10 librados: watch_flush exit 2024-03-25T19:41:30.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.126+0000 7f51d8d4d640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-25T19:41:30.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.126+0000 7f51d8d4d640 10 client.4479.objecter successfully canceled tick 2024-03-25T19:41:30.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.126+0000 7f51d8d4d640 1 -- 172.21.15.107:0/1321489867 >> v1:172.21.15.195:6800/2984408421 conn(0x7f51b406df70 legacy=0x7f51b4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:30.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.126+0000 7f51d8d4d640 10 monclient: shutdown 2024-03-25T19:41:30.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.126+0000 7f51d8d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:30.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.126+0000 7f51d8d4d640 1 -- 172.21.15.107:0/1321489867 >> v1:172.21.15.107:6789/0 conn(0x7f51d414be90 legacy=0x7f51d41e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:30.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.126+0000 7f51d8d4d640 1 -- 172.21.15.107:0/1321489867 shutdown_connections 2024-03-25T19:41:30.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.126+0000 7f51d8d4d640 1 -- 172.21.15.107:0/1321489867 >> 172.21.15.107:0/1321489867 conn(0x7f51d40617a0 msgr2=0x7f51d40b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:30.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.127+0000 7f51d8d4d640 1 -- 172.21.15.107:0/1321489867 shutdown_connections 2024-03-25T19:41:30.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.127+0000 7f51d8d4d640 1 -- 172.21.15.107:0/1321489867 wait complete. 2024-03-25T19:41:30.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:30.127+0000 7f51d8d4d640 1 librados: shutdown 2024-03-25T19:41:30.138 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:36.945 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:37.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.060+0000 7f8cd8f97640 10 monclient: get_monmap_and_config 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.060+0000 7f8cd8f97640 10 monclient: build_initial_monmap 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.060+0000 7f8cd8f97640 10 monclient: monmap: 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:41:37.061280+0000 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:41:37.061280+0000 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:41:37.060 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:37.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.061+0000 7f8cd8f97640 1 Processor -- start 2024-03-25T19:41:37.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.061+0000 7f8cd8f97640 1 -- start start 2024-03-25T19:41:37.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.061+0000 7f8cd8f97640 10 monclient: init 2024-03-25T19:41:37.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.061+0000 7f8cd8f97640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:37.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.061+0000 7f8cd8f97640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:37.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.061+0000 7f8cd8f97640 10 monclient(hunting): picked mon.noname-a con 0x7f8cd40a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:37.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.062+0000 7f8cd8f97640 10 monclient(hunting): picked mon.noname-b con 0x7f8cd413f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:37.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.062+0000 7f8cd8f97640 10 monclient(hunting): picked mon.noname-c con 0x7f8cd41430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:37.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.062+0000 7f8cd8f97640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd4142520 con 0x7f8cd40a98f0 2024-03-25T19:41:37.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.062+0000 7f8cd8f97640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd40b28a0 con 0x7f8cd41430c0 2024-03-25T19:41:37.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.062+0000 7f8cd8f97640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd40b2a30 con 0x7f8cd413f710 2024-03-25T19:41:37.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.062+0000 7f8cd8f97640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:37.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.062+0000 7f8cd8f97640 10 monclient(hunting): authenticate will time out at 4889.402830s 2024-03-25T19:41:37.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.067+0000 7f8cd2d9d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f8cd41430c0 0x7f8cd414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:57766/0 (socket says 172.21.15.107:57766) 2024-03-25T19:41:37.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.067+0000 7f8cd2d9d640 1 -- 172.21.15.107:0/3936740433 learned_addr learned my addr 172.21.15.107:0/3936740433 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:37.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.082+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392930632 0 0) 0x7f8cd4142520 con 0x7f8cd40a98f0 2024-03-25T19:41:37.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.082+0000 7f8cd1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.082+0000 7f8cd1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.082+0000 7f8cd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.082+0000 7f8cd1d9b640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:41:37.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.082+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc0003520 con 0x7f8cd40a98f0 2024-03-25T19:41:37.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.085+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416387431 0 0) 0x7f8cd40b28a0 con 0x7f8cd41430c0 2024-03-25T19:41:37.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.085+0000 7f8cd1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.085+0000 7f8cd1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.085+0000 7f8cd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.085+0000 7f8cd1d9b640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:41:37.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.085+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd4142520 con 0x7f8cd41430c0 2024-03-25T19:41:37.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.091+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827372433 0 0) 0x7f8cd40b2a30 con 0x7f8cd413f710 2024-03-25T19:41:37.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.091+0000 7f8cd1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.091+0000 7f8cd1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.091+0000 7f8cd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.091+0000 7f8cd1d9b640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:41:37.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.091+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd40b28a0 con 0x7f8cd413f710 2024-03-25T19:41:37.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3800072216 0 0) 0x7f8cd4142520 con 0x7f8cd41430c0 2024-03-25T19:41:37.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 >> v1:172.21.15.107:6790/0 conn(0x7f8cd413f710 legacy=0x7f8cd41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 >> v1:172.21.15.107:6789/0 conn(0x7f8cd40a98f0 legacy=0x7f8cd40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 1 monclient: found mon.noname-c 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd40b44c0 con 0x7f8cd41430c0 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 10 monclient: _finish_auth 0 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8ccc0019f0 con 0x7f8cd41430c0 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd1d9b640 10 monclient: dump: 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:37.096 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:37.097 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:37.097 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:37.097 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:37.097 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:37.097 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:37.097 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:37.097 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:37.097 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:37.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.095+0000 7f8cd8f97640 5 monclient: authenticate success, global_id 4471 2024-03-25T19:41:37.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.096+0000 7f8cd8f97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:37.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ccc001b50 con 0x7f8cd41430c0 2024-03-25T19:41:37.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:37.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd1d9b640 1 -- 172.21.15.107:0/3936740433 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8ccc001f60 con 0x7f8cd41430c0 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd1d9b640 10 monclient: dump: 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd8f97640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:37.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd8f97640 4 set_mon_vals no callback set 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd8f97640 10 monclient: shutdown 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd8f97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd8f97640 1 -- 172.21.15.107:0/3936740433 >> v1:172.21.15.195:6789/0 conn(0x7f8cd41430c0 legacy=0x7f8cd414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd8f97640 1 -- 172.21.15.107:0/3936740433 shutdown_connections 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.107+0000 7f8cd8f97640 1 -- 172.21.15.107:0/3936740433 >> 172.21.15.107:0/3936740433 conn(0x7f8cd40b7ea0 msgr2=0x7f8cd40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.108+0000 7f8cd8f97640 1 -- 172.21.15.107:0/3936740433 shutdown_connections 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.108+0000 7f8cd8f97640 1 -- 172.21.15.107:0/3936740433 wait complete. 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.108+0000 7f8cd8f97640 10 monclient: build_initial_monmap 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.108+0000 7f8cd8f97640 10 monclient: monmap: 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:41:37.110 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 1 librados: starting msgr at 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 1 librados: starting objecter 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 1 Processor -- start 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 1 -- start start 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 1 librados: setting wanted keys 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 1 librados: calling monclient init 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 10 monclient: init 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:37.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 10 monclient(hunting): picked mon.noname-b con 0x7f8cd40a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:37.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 10 monclient(hunting): picked mon.noname-c con 0x7f8cd413f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:37.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.109+0000 7f8cd8f97640 10 monclient(hunting): picked mon.noname-a con 0x7f8cd41430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:37.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.110+0000 7f8cd8f97640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd4137ce0 con 0x7f8cd41430c0 2024-03-25T19:41:37.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.110+0000 7f8cd8f97640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd41396e0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.110+0000 7f8cd8f97640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd41399c0 con 0x7f8cd413f710 2024-03-25T19:41:37.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.110+0000 7f8cd8f97640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:37.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.110+0000 7f8cd8f97640 10 monclient(hunting): authenticate will time out at 4889.450854s 2024-03-25T19:41:37.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.117+0000 7f8ccbfff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f8cd413f710 0x7f8cd4138320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45774/0 (socket says 172.21.15.107:45774) 2024-03-25T19:41:37.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.117+0000 7f8ccbfff640 1 -- 172.21.15.107:0/398346334 learned_addr learned my addr 172.21.15.107:0/398346334 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:37.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.117+0000 7f8ccaffd640 10 client.?.objecter ms_handle_connect 0x7f8cd413f710 2024-03-25T19:41:37.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.117+0000 7f8ccaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:37.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.120+0000 7f8ccaffd640 10 client.?.objecter ms_handle_connect 0x7f8cd41430c0 2024-03-25T19:41:37.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.120+0000 7f8ccaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:37.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.120+0000 7f8ccaffd640 10 client.?.objecter ms_handle_connect 0x7f8cd40a98f0 2024-03-25T19:41:37.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.120+0000 7f8ccaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:37.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.128+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444465782 0 0) 0x7f8cd41396e0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.129+0000 7f8ccaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.129+0000 7f8ccaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.129+0000 7f8ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.129+0000 7f8ccaffd640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:41:37.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.129+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb4004eb0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.130+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321521879 0 0) 0x7f8cd41399c0 con 0x7f8cd413f710 2024-03-25T19:41:37.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.130+0000 7f8ccaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.130+0000 7f8ccaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.130+0000 7f8ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.130+0000 7f8ccaffd640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:41:37.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.130+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd41396e0 con 0x7f8cd413f710 2024-03-25T19:41:37.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.134+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444422827 0 0) 0x7f8cd4137ce0 con 0x7f8cd41430c0 2024-03-25T19:41:37.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.134+0000 7f8ccaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:37.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.134+0000 7f8ccaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:37.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.134+0000 7f8ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:37.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.134+0000 7f8ccaffd640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:41:37.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.134+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd41399c0 con 0x7f8cd41430c0 2024-03-25T19:41:37.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.139+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 520672185 0 0) 0x7f8cb4004eb0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.139+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cd4137ce0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.139+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8cc4001950 con 0x7f8cd40a98f0 2024-03-25T19:41:37.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.139+0000 7f8ccaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:37.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.139+0000 7f8ccaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:37.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.139+0000 7f8ccaffd640 10 monclient(hunting): dump: 2024-03-25T19:41:37.139 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:37.139 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:37.139 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:37.140 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:37.140 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:37.140 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:37.140 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:37.140 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:37.140 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:37.140 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:37.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.139+0000 7f8ccaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:37.144 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.145+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1197026448 0 0) 0x7f8cd41396e0 con 0x7f8cd413f710 2024-03-25T19:41:37.144 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.145+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb4004eb0 con 0x7f8cd413f710 2024-03-25T19:41:37.144 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.145+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8cbc0019f0 con 0x7f8cd413f710 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.145+0000 7f8ccaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.145+0000 7f8ccaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.145+0000 7f8ccaffd640 10 monclient(hunting): dump: 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:37.145 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:37.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 990811058 0 0) 0x7f8cd4137ce0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 >> v1:172.21.15.107:6790/0 conn(0x7f8cd413f710 legacy=0x7f8cd4138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 >> v1:172.21.15.107:6789/0 conn(0x7f8cd41430c0 legacy=0x7f8cd41375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8ccaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:37.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8ccaffd640 1 monclient: found mon.b 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8ccaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8ccaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd40b0fb0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8ccaffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8ccaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8ccaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8cd8f97640 5 monclient: authenticate success, global_id 4474 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8cd8f97640 10 monclient: _renew_subs 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8cd8f97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8cd8f97640 1 -- 172.21.15.107:0/398346334 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd40afdc0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8cd8f97640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8cd8f97640 10 monclient: _renew_subs 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8cd8f97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8cd8f97640 1 -- 172.21.15.107:0/398346334 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd40b02a0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.155+0000 7f8cd8f97640 1 librados: init done 2024-03-25T19:41:37.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.156+0000 7f8cd8f97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:37.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.156+0000 7f8cd8f97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:37.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.156+0000 7f8cd8f97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:37.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.156+0000 7f8cd8f97640 1 -- 172.21.15.107:0/398346334 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd40627b0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.426+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cc4003610 con 0x7f8cd40a98f0 2024-03-25T19:41:37.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.426+0000 7f8ccaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:37.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.426+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8cc4001d80 con 0x7f8cd40a98f0 2024-03-25T19:41:37.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.426+0000 7f8ccaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:37.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.426+0000 7f8ccaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:37.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.426+0000 7f8ccaffd640 10 monclient: dump: 2024-03-25T19:41:37.426 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:37.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.426+0000 7f8cd159a640 4 set_mon_vals no callback set 2024-03-25T19:41:37.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.427+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f8cc40197d0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.428+0000 7f8ccbfff640 10 monclient: get_auth_request con 0x7f8cb406df20 auth_method 0 2024-03-25T19:41:37.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.428+0000 7f8ccaffd640 10 client.4474.objecter ms_handle_connect 0x7f8cb406df20 2024-03-25T19:41:37.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.432+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f8cc4053510 con 0x7f8cd40a98f0 2024-03-25T19:41:37.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.432+0000 7f8ccaffd640 10 client.4474.objecter ms_dispatch 0x7f8cd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:37.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.432+0000 7f8ccaffd640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:37.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.432+0000 7f8ccaffd640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:37.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.432+0000 7f8ccaffd640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-25T19:41:37.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.433+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8cc404ee50 con 0x7f8cd40a98f0 2024-03-25T19:41:37.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.433+0000 7f8ccaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:37.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.433+0000 7f8ccaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:37.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.661+0000 7f8cd8f97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:37.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.661+0000 7f8cd8f97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:37.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.661+0000 7f8cd8f97640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:37.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.661+0000 7f8cd8f97640 1 -- 172.21.15.107:0/398346334 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cd40b71a0 con 0x7f8cd40a98f0 2024-03-25T19:41:37.684 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.684+0000 7f8ccaffd640 1 -- 172.21.15.107:0/398346334 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f8cc4050f50 con 0x7f8cd40a98f0 2024-03-25T19:41:37.684 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.684+0000 7f8ccaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:37.684 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.684+0000 7f8ccaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:37.684 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:41:37.684 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:37.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 10 librados: watch_flush enter 2024-03-25T19:41:37.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 10 librados: watch_flush exit 2024-03-25T19:41:37.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-25T19:41:37.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 10 client.4474.objecter successfully canceled tick 2024-03-25T19:41:37.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 1 -- 172.21.15.107:0/398346334 >> v1:172.21.15.195:6800/2984408421 conn(0x7f8cb406df20 legacy=0x7f8cb40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 10 monclient: shutdown 2024-03-25T19:41:37.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:37.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 1 -- 172.21.15.107:0/398346334 >> v1:172.21.15.195:6789/0 conn(0x7f8cd40a98f0 legacy=0x7f8cd41da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:37.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 1 -- 172.21.15.107:0/398346334 shutdown_connections 2024-03-25T19:41:37.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 1 -- 172.21.15.107:0/398346334 >> 172.21.15.107:0/398346334 conn(0x7f8cd40b7ea0 msgr2=0x7f8cd4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:37.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 1 -- 172.21.15.107:0/398346334 shutdown_connections 2024-03-25T19:41:37.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 1 -- 172.21.15.107:0/398346334 wait complete. 2024-03-25T19:41:37.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:37.686+0000 7f8cd8f97640 1 librados: shutdown 2024-03-25T19:41:37.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:44.506 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:44.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.627+0000 7fa92f57b640 10 monclient: get_monmap_and_config 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.627+0000 7fa92f57b640 10 monclient: build_initial_monmap 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.627+0000 7fa92f57b640 10 monclient: monmap: 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:41:44.628090+0000 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:41:44.628090+0000 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:41:44.627 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:44.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.627+0000 7fa92f57b640 1 Processor -- start 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.628+0000 7fa92f57b640 1 -- start start 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.628+0000 7fa92f57b640 10 monclient: init 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.628+0000 7fa92f57b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.628+0000 7fa92f57b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.628+0000 7fa92f57b640 10 monclient(hunting): picked mon.noname-c con 0x7fa9280a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.628+0000 7fa92f57b640 10 monclient(hunting): picked mon.noname-b con 0x7fa92813f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.628+0000 7fa92f57b640 10 monclient(hunting): picked mon.noname-a con 0x7fa9281430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.628+0000 7fa92f57b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa928142520 con 0x7fa9281430c0 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.628+0000 7fa92f57b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280b28a0 con 0x7fa9280a98f0 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.629+0000 7fa92f57b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280b2a30 con 0x7fa92813f710 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.629+0000 7fa92f57b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:44.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.629+0000 7fa92f57b640 10 monclient(hunting): authenticate will time out at 4896.969762s 2024-03-25T19:41:44.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.639+0000 7fa92cb16640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa92813f710 0x7fa9281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47784/0 (socket says 172.21.15.107:47784) 2024-03-25T19:41:44.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.639+0000 7fa92cb16640 1 -- 172.21.15.107:0/1945737374 learned_addr learned my addr 172.21.15.107:0/1945737374 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:44.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.658+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998137517 0 0) 0x7fa928142520 con 0x7fa9281430c0 2024-03-25T19:41:44.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.658+0000 7fa91ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.658+0000 7fa91ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.658+0000 7fa91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.658+0000 7fa91ffff640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:41:44.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.658+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa914003520 con 0x7fa9281430c0 2024-03-25T19:41:44.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.659+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570943423 0 0) 0x7fa9280b2a30 con 0x7fa92813f710 2024-03-25T19:41:44.659 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.659+0000 7fa91ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.659 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.659+0000 7fa91ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.659 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.659+0000 7fa91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.659 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.659+0000 7fa91ffff640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:41:44.659 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.659+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa928142520 con 0x7fa92813f710 2024-03-25T19:41:44.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.667+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165330387 0 0) 0x7fa9280b28a0 con 0x7fa9280a98f0 2024-03-25T19:41:44.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.667+0000 7fa91ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.667+0000 7fa91ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.667+0000 7fa91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.667+0000 7fa91ffff640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:41:44.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.667+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9280b2a30 con 0x7fa9280a98f0 2024-03-25T19:41:44.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3715912021 0 0) 0x7fa928142520 con 0x7fa92813f710 2024-03-25T19:41:44.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 >> v1:172.21.15.195:6789/0 conn(0x7fa9280a98f0 legacy=0x7fa9280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 >> v1:172.21.15.107:6789/0 conn(0x7fa9281430c0 legacy=0x7fa92814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:44.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 1 monclient: found mon.noname-b 2024-03-25T19:41:44.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:44.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:41:44.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9280b44c0 con 0x7fa92813f710 2024-03-25T19:41:44.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 10 monclient: _finish_auth 0 2024-03-25T19:41:44.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fa918001670 con 0x7fa92813f710 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa91ffff640 10 monclient: dump: 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:44.675 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:44.676 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:44.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa92f57b640 5 monclient: authenticate success, global_id 4490 2024-03-25T19:41:44.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.674+0000 7fa92f57b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:44.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.699+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9180017d0 con 0x7fa92813f710 2024-03-25T19:41:44.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.699+0000 7fa91ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:44.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.699+0000 7fa91ffff640 1 -- 172.21.15.107:0/1945737374 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fa918001be0 con 0x7fa92813f710 2024-03-25T19:41:44.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.699+0000 7fa91ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.699+0000 7fa91ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:44.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.699+0000 7fa91ffff640 10 monclient: dump: 2024-03-25T19:41:44.699 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:44.699 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:44.699 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:44.699 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:44.700 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.700 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:44.700 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:44.700 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:44.700 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:44.700 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:44.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.699+0000 7fa92f57b640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:44.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.699+0000 7fa92f57b640 4 set_mon_vals no callback set 2024-03-25T19:41:44.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.700+0000 7fa92f57b640 10 monclient: shutdown 2024-03-25T19:41:44.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.700+0000 7fa92f57b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:44.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.700+0000 7fa92f57b640 1 -- 172.21.15.107:0/1945737374 >> v1:172.21.15.107:6790/0 conn(0x7fa92813f710 legacy=0x7fa9281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.700+0000 7fa92f57b640 1 -- 172.21.15.107:0/1945737374 shutdown_connections 2024-03-25T19:41:44.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.700+0000 7fa92f57b640 1 -- 172.21.15.107:0/1945737374 >> 172.21.15.107:0/1945737374 conn(0x7fa9280b7ea0 msgr2=0x7fa9280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:44.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.701+0000 7fa92f57b640 1 -- 172.21.15.107:0/1945737374 shutdown_connections 2024-03-25T19:41:44.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.701+0000 7fa92f57b640 1 -- 172.21.15.107:0/1945737374 wait complete. 2024-03-25T19:41:44.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.701+0000 7fa92f57b640 10 monclient: build_initial_monmap 2024-03-25T19:41:44.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.701+0000 7fa92f57b640 10 monclient: monmap: 2024-03-25T19:41:44.702 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:44.702 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:44.702 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:41:44.702 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:41:44.702 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:44.702 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:41:44.702 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:44.702 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:41:44.702 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:41:44.702 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:44.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.702+0000 7fa92f57b640 1 librados: starting msgr at 2024-03-25T19:41:44.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.702+0000 7fa92f57b640 1 librados: starting objecter 2024-03-25T19:41:44.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.702+0000 7fa92f57b640 1 Processor -- start 2024-03-25T19:41:44.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.702+0000 7fa92f57b640 1 -- start start 2024-03-25T19:41:44.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.702+0000 7fa92f57b640 1 librados: setting wanted keys 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.702+0000 7fa92f57b640 1 librados: calling monclient init 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.702+0000 7fa92f57b640 10 monclient: init 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.703+0000 7fa92f57b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.703+0000 7fa92f57b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.703+0000 7fa92f57b640 10 monclient(hunting): picked mon.noname-b con 0x7fa9280a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.703+0000 7fa92f57b640 10 monclient(hunting): picked mon.noname-a con 0x7fa92813f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.703+0000 7fa92f57b640 10 monclient(hunting): picked mon.noname-c con 0x7fa9281430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.703+0000 7fa92f57b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa928138ed0 con 0x7fa92813f710 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.703+0000 7fa92f57b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280abc80 con 0x7fa9280a98f0 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.703+0000 7fa92f57b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280abf60 con 0x7fa9281430c0 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.703+0000 7fa92f57b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:44.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.703+0000 7fa92f57b640 10 monclient(hunting): authenticate will time out at 4897.044381s 2024-03-25T19:41:44.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.709+0000 7fa92d317640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fa9280a98f0 0x7fa9281387c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:54834/0 (socket says 172.21.15.107:54834) 2024-03-25T19:41:44.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.709+0000 7fa92d317640 1 -- 172.21.15.107:0/1786995550 learned_addr learned my addr 172.21.15.107:0/1786995550 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:44.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.710+0000 7fa91dffb640 10 client.?.objecter ms_handle_connect 0x7fa9280a98f0 2024-03-25T19:41:44.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.710+0000 7fa91dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:44.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.719+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372157613 0 0) 0x7fa9280abc80 con 0x7fa9280a98f0 2024-03-25T19:41:44.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.719+0000 7fa91dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.719+0000 7fa91dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.719+0000 7fa91dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.719+0000 7fa91dffb640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:41:44.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.719+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa90c004eb0 con 0x7fa9280a98f0 2024-03-25T19:41:44.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.725+0000 7fa91dffb640 10 client.?.objecter ms_handle_connect 0x7fa9281430c0 2024-03-25T19:41:44.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.725+0000 7fa91dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:44.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.729+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2231231214 0 0) 0x7fa90c004eb0 con 0x7fa9280a98f0 2024-03-25T19:41:44.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.729+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9280abc80 con 0x7fa9280a98f0 2024-03-25T19:41:44.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.729+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fa924001a10 con 0x7fa9280a98f0 2024-03-25T19:41:44.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.729+0000 7fa91dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.729+0000 7fa91dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:44.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.729+0000 7fa91dffb640 10 monclient(hunting): dump: 2024-03-25T19:41:44.729 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:44.729 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:44.730 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:44.730 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:44.730 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.730 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:44.730 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:44.730 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:44.730 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:44.730 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:44.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.729+0000 7fa91dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:44.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.731+0000 7fa91dffb640 10 client.?.objecter ms_handle_connect 0x7fa92813f710 2024-03-25T19:41:44.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.731+0000 7fa91dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:44.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.739+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2777921716 0 0) 0x7fa9280abc80 con 0x7fa9280a98f0 2024-03-25T19:41:44.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 >> v1:172.21.15.107:6790/0 conn(0x7fa9281430c0 legacy=0x7fa9281e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 >> v1:172.21.15.107:6789/0 conn(0x7fa92813f710 legacy=0x7fa9280ac660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa91dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:44.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa91dffb640 1 monclient: found mon.b 2024-03-25T19:41:44.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa91dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:44.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa91dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:44.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa928139170 con 0x7fa9280a98f0 2024-03-25T19:41:44.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa91dffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:44.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa91dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:44.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa91dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:44.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa92f57b640 5 monclient: authenticate success, global_id 4480 2024-03-25T19:41:44.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa92f57b640 10 monclient: _renew_subs 2024-03-25T19:41:44.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa92f57b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:44.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa92f57b640 1 -- 172.21.15.107:0/1786995550 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9280acd70 con 0x7fa9280a98f0 2024-03-25T19:41:44.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa92f57b640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:44.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa92f57b640 10 monclient: _renew_subs 2024-03-25T19:41:44.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa92f57b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:44.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa92f57b640 1 -- 172.21.15.107:0/1786995550 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa928136c30 con 0x7fa9280a98f0 2024-03-25T19:41:44.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.740+0000 7fa92f57b640 1 librados: init done 2024-03-25T19:41:44.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.741+0000 7fa92f57b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:44.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.741+0000 7fa92f57b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:44.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.741+0000 7fa92f57b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:44.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:44.741+0000 7fa92f57b640 1 -- 172.21.15.107:0/1786995550 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa928136690 con 0x7fa9280a98f0 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.013+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa924001cf0 con 0x7fa9280a98f0 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.013+0000 7fa91dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.013+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fa924003a60 con 0x7fa9280a98f0 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.013+0000 7fa91dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.013+0000 7fa91dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.013+0000 7fa91dffb640 10 monclient: dump: 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:45.013 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:45.014 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:45.014 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:45.014 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:45.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.013+0000 7fa91f7fe640 4 set_mon_vals no callback set 2024-03-25T19:41:45.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.014+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fa924003c80 con 0x7fa9280a98f0 2024-03-25T19:41:45.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.014+0000 7fa92cb16640 10 monclient: get_auth_request con 0x7fa90c06dc70 auth_method 0 2024-03-25T19:41:45.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.015+0000 7fa91dffb640 10 client.4480.objecter ms_handle_connect 0x7fa90c06dc70 2024-03-25T19:41:45.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.018+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fa9240548d0 con 0x7fa9280a98f0 2024-03-25T19:41:45.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.018+0000 7fa91dffb640 10 client.4480.objecter ms_dispatch 0x7fa928000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:45.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.018+0000 7fa91dffb640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:45.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.018+0000 7fa91dffb640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:45.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.018+0000 7fa91dffb640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.018+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9240503e0 con 0x7fa9280a98f0 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.018+0000 7fa91dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:45.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.018+0000 7fa91dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:45.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.262+0000 7fa92f57b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:45.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.262+0000 7fa92f57b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:45.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.262+0000 7fa92f57b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:45.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.262+0000 7fa92f57b640 1 -- 172.21.15.107:0/1786995550 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9280627b0 con 0x7fa9280a98f0 2024-03-25T19:41:45.304 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:06:15.620 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.312 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:06:15.629 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.348 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:06:15.665 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.360 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:06:15.676 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.366 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:06:15.683 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.384 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:06:15.671 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.390 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:06:15.707 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.410 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:06:15.697 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.418+0000 7fa91dffb640 1 -- 172.21.15.107:0/1786995550 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fa924052310 con 0x7fa9280a98f0 2024-03-25T19:41:45.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.418+0000 7fa91dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:45.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.418+0000 7fa91dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:45.418 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:41:45.418 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:45.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 10 librados: watch_flush enter 2024-03-25T19:41:45.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 10 librados: watch_flush exit 2024-03-25T19:41:45.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-25T19:41:45.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 10 client.4480.objecter successfully canceled tick 2024-03-25T19:41:45.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 1 -- 172.21.15.107:0/1786995550 >> v1:172.21.15.195:6800/2984408421 conn(0x7fa90c06dc70 legacy=0x7fa90c070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 10 monclient: shutdown 2024-03-25T19:41:45.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:45.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 1 -- 172.21.15.107:0/1786995550 >> v1:172.21.15.195:6789/0 conn(0x7fa9280a98f0 legacy=0x7fa9281387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 1 -- 172.21.15.107:0/1786995550 shutdown_connections 2024-03-25T19:41:45.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 1 -- 172.21.15.107:0/1786995550 >> 172.21.15.107:0/1786995550 conn(0x7fa9280b7ea0 msgr2=0x7fa928074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:45.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 1 -- 172.21.15.107:0/1786995550 shutdown_connections 2024-03-25T19:41:45.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 1 -- 172.21.15.107:0/1786995550 wait complete. 2024-03-25T19:41:45.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:45.420+0000 7fa92f57b640 1 librados: shutdown 2024-03-25T19:41:45.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:45.435 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:06:15.722 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:52.239 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.336+0000 7fb829e96640 10 monclient: get_monmap_and_config 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.336+0000 7fb829e96640 10 monclient: build_initial_monmap 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.336+0000 7fb829e96640 10 monclient: monmap: 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:41:52.337406+0000 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:41:52.337406+0000 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:41:52.336 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:41:52.337 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:52.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 1 Processor -- start 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 1 -- start start 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 10 monclient: init 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 10 monclient(hunting): picked mon.noname-b con 0x7fb8240b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 10 monclient(hunting): picked mon.noname-a con 0x7fb8240b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 10 monclient(hunting): picked mon.noname-c con 0x7fb82414be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8240af940 con 0x7fb8240b1730 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8240b0a70 con 0x7fb82414be90 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8240b0d50 con 0x7fb8240b4d60 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:52.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.337+0000 7fb829e96640 10 monclient(hunting): authenticate will time out at 4904.678720s 2024-03-25T19:41:52.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.351+0000 7fb8237fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb8240b4d60 0x7fb82413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43076/0 (socket says 172.21.15.107:43076) 2024-03-25T19:41:52.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.351+0000 7fb8237fe640 1 -- 172.21.15.107:0/2816206204 learned_addr learned my addr 172.21.15.107:0/2816206204 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:52.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.357+0000 7fb8227fc640 1 -- 172.21.15.107:0/2816206204 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188856042 0 0) 0x7fb8240b0d50 con 0x7fb8240b4d60 2024-03-25T19:41:52.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.357+0000 7fb8227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:52.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.357+0000 7fb8227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:52.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.357+0000 7fb8227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:52.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.357+0000 7fb8227fc640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:41:52.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.357+0000 7fb8227fc640 1 -- 172.21.15.107:0/2816206204 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb80c003520 con 0x7fb8240b4d60 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.385+0000 7fb8227fc640 1 -- 172.21.15.107:0/2816206204 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1000074288 0 0) 0x7fb80c003520 con 0x7fb8240b4d60 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.385+0000 7fb8227fc640 1 -- 172.21.15.107:0/2816206204 >> v1:172.21.15.195:6789/0 conn(0x7fb82414be90 legacy=0x7fb82414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 1 -- 172.21.15.107:0/2816206204 >> v1:172.21.15.107:6789/0 conn(0x7fb8240b1730 legacy=0x7fb8240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 1 monclient: found mon.noname-b 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 1 -- 172.21.15.107:0/2816206204 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8240b1030 con 0x7fb8240b4d60 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:52.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 1 -- 172.21.15.107:0/2816206204 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb814001690 con 0x7fb8240b4d60 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb8227fc640 10 monclient: dump: 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:52.387 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:52.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb829e96640 5 monclient: authenticate success, global_id 4496 2024-03-25T19:41:52.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.386+0000 7fb829e96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:52.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.418+0000 7fb8227fc640 1 -- 172.21.15.107:0/2816206204 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb814001860 con 0x7fb8240b4d60 2024-03-25T19:41:52.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.418+0000 7fb8227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:52.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.418+0000 7fb8227fc640 1 -- 172.21.15.107:0/2816206204 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb814001cb0 con 0x7fb8240b4d60 2024-03-25T19:41:52.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.418+0000 7fb829e96640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:52.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.418+0000 7fb829e96640 4 set_mon_vals no callback set 2024-03-25T19:41:52.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.418+0000 7fb8227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:52.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.418+0000 7fb8227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:52.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.418+0000 7fb8227fc640 10 monclient: dump: 2024-03-25T19:41:52.419 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:52.419 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:52.419 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:52.419 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:52.419 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:52.419 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:52.419 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:52.419 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:52.419 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:52.419 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:52.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.419+0000 7fb829e96640 10 monclient: shutdown 2024-03-25T19:41:52.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.419+0000 7fb829e96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:52.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.419+0000 7fb829e96640 1 -- 172.21.15.107:0/2816206204 >> v1:172.21.15.107:6790/0 conn(0x7fb8240b4d60 legacy=0x7fb82413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.419+0000 7fb829e96640 1 -- 172.21.15.107:0/2816206204 shutdown_connections 2024-03-25T19:41:52.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.419+0000 7fb829e96640 1 -- 172.21.15.107:0/2816206204 >> 172.21.15.107:0/2816206204 conn(0x7fb8240617a0 msgr2=0x7fb824061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.419+0000 7fb829e96640 1 -- 172.21.15.107:0/2816206204 shutdown_connections 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.419+0000 7fb829e96640 1 -- 172.21.15.107:0/2816206204 wait complete. 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.419+0000 7fb829e96640 10 monclient: build_initial_monmap 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.419+0000 7fb829e96640 10 monclient: monmap: 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:41:52.423 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 1 librados: starting msgr at 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 1 librados: starting objecter 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 1 Processor -- start 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 1 -- start start 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 1 librados: setting wanted keys 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 1 librados: calling monclient init 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 10 monclient: init 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 10 monclient(hunting): picked mon.noname-a con 0x7fb8240b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 10 monclient(hunting): picked mon.noname-c con 0x7fb8240b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:41:52.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 10 monclient(hunting): picked mon.noname-b con 0x7fb82414be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:41:52.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.420+0000 7fb829e96640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb824136270 con 0x7fb8240b1730 2024-03-25T19:41:52.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.421+0000 7fb829e96640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb824135d50 con 0x7fb82414be90 2024-03-25T19:41:52.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.421+0000 7fb829e96640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb824136e00 con 0x7fb8240b4d60 2024-03-25T19:41:52.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.421+0000 7fb829e96640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:52.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.421+0000 7fb829e96640 10 monclient(hunting): authenticate will time out at 4904.761810s 2024-03-25T19:41:52.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.426+0000 7fb823fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb82414be90 0x7fb8241e2f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:43520/0 (socket says 172.21.15.107:43520) 2024-03-25T19:41:52.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.426+0000 7fb823fff640 1 -- 172.21.15.107:0/3403800785 learned_addr learned my addr 172.21.15.107:0/3403800785 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:41:52.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.428+0000 7fb80bfff640 10 client.?.objecter ms_handle_connect 0x7fb8240b1730 2024-03-25T19:41:52.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.428+0000 7fb80bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:52.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.429+0000 7fb80bfff640 10 client.?.objecter ms_handle_connect 0x7fb82414be90 2024-03-25T19:41:52.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.429+0000 7fb80bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:52.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.436+0000 7fb80bfff640 10 client.?.objecter ms_handle_connect 0x7fb8240b4d60 2024-03-25T19:41:52.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.436+0000 7fb80bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:52.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.445+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158205126 0 0) 0x7fb824135d50 con 0x7fb82414be90 2024-03-25T19:41:52.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.445+0000 7fb80bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:52.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.445+0000 7fb80bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:52.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.445+0000 7fb80bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:52.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.445+0000 7fb80bfff640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:41:52.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.445+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb810004eb0 con 0x7fb82414be90 2024-03-25T19:41:52.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.446+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445838603 0 0) 0x7fb824136270 con 0x7fb8240b1730 2024-03-25T19:41:52.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.446+0000 7fb80bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:52.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.446+0000 7fb80bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:52.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.446+0000 7fb80bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:52.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.446+0000 7fb80bfff640 10 monclient(hunting): my global_id is 4494 2024-03-25T19:41:52.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.446+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb824135d50 con 0x7fb8240b1730 2024-03-25T19:41:52.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.455+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3734517755 0 0) 0x7fb810004eb0 con 0x7fb82414be90 2024-03-25T19:41:52.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.455+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb824136270 con 0x7fb82414be90 2024-03-25T19:41:52.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.455+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb81c001c90 con 0x7fb82414be90 2024-03-25T19:41:52.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.455+0000 7fb80bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:52.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.455+0000 7fb80bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:52.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.455+0000 7fb80bfff640 10 monclient(hunting): dump: 2024-03-25T19:41:52.455 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:52.455 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:52.455 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:52.455 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:52.455 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:52.456 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:52.456 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:52.456 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:52.456 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:52.456 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:52.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.455+0000 7fb80bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:52.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.460+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145547760 0 0) 0x7fb824136e00 con 0x7fb8240b4d60 2024-03-25T19:41:52.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.460+0000 7fb80bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:52.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.460+0000 7fb80bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:52.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.460+0000 7fb80bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:52.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.460+0000 7fb80bfff640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:41:52.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.460+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb810004eb0 con 0x7fb8240b4d60 2024-03-25T19:41:52.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.471+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2700185552 0 0) 0x7fb824135d50 con 0x7fb8240b1730 2024-03-25T19:41:52.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.471+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb824136e00 con 0x7fb8240b1730 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.471+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb814001b70 con 0x7fb8240b1730 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.471+0000 7fb80bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.471+0000 7fb80bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.471+0000 7fb80bfff640 10 monclient(hunting): dump: 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:52.471 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:52.481 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1025343013 0 0) 0x7fb824136270 con 0x7fb82414be90 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 >> v1:172.21.15.107:6790/0 conn(0x7fb8240b4d60 legacy=0x7fb8241dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 >> v1:172.21.15.107:6789/0 conn(0x7fb8240b1730 legacy=0x7fb82412fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb80bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb80bfff640 1 monclient: found mon.b 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb80bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb80bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb824136f90 con 0x7fb82414be90 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb80bfff640 10 monclient: _finish_auth 0 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb80bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb80bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb829e96640 5 monclient: authenticate success, global_id 4486 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb829e96640 10 monclient: _renew_subs 2024-03-25T19:41:52.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb829e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:52.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.481+0000 7fb829e96640 1 -- 172.21.15.107:0/3403800785 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb824130600 con 0x7fb82414be90 2024-03-25T19:41:52.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.482+0000 7fb829e96640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:52.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.482+0000 7fb829e96640 10 monclient: _renew_subs 2024-03-25T19:41:52.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.482+0000 7fb829e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:52.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.482+0000 7fb829e96640 1 -- 172.21.15.107:0/3403800785 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8241366b0 con 0x7fb82414be90 2024-03-25T19:41:52.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.482+0000 7fb829e96640 1 librados: init done 2024-03-25T19:41:52.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.483+0000 7fb829e96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:52.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.483+0000 7fb829e96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:52.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.483+0000 7fb829e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:52.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.483+0000 7fb829e96640 1 -- 172.21.15.107:0/3403800785 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8241307f0 con 0x7fb82414be90 2024-03-25T19:41:52.785 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.785+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb81c003990 con 0x7fb82414be90 2024-03-25T19:41:52.785 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.785+0000 7fb80bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:52.785 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.785+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb81c003ce0 con 0x7fb82414be90 2024-03-25T19:41:52.785 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.785+0000 7fb80bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.785+0000 7fb80bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.785+0000 7fb80bfff640 10 monclient: dump: 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:41:52.786 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.785+0000 7fb821ffb640 4 set_mon_vals no callback set 2024-03-25T19:41:52.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.786+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fb81c003f00 con 0x7fb82414be90 2024-03-25T19:41:52.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.787+0000 7fb8237fe640 10 monclient: get_auth_request con 0x7fb81006dfb0 auth_method 0 2024-03-25T19:41:52.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.787+0000 7fb80bfff640 10 client.4486.objecter ms_handle_connect 0x7fb81006dfb0 2024-03-25T19:41:52.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.791+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fb81c054a50 con 0x7fb82414be90 2024-03-25T19:41:52.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.791+0000 7fb80bfff640 10 client.4486.objecter ms_dispatch 0x7fb824000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:52.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.791+0000 7fb80bfff640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:52.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.791+0000 7fb80bfff640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:52.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.791+0000 7fb80bfff640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-25T19:41:52.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.791+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb81c050560 con 0x7fb82414be90 2024-03-25T19:41:52.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.791+0000 7fb80bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:52.791 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:52.791+0000 7fb80bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:53.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.044+0000 7fb829e96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:53.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.044+0000 7fb829e96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:53.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.044+0000 7fb829e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:41:53.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.044+0000 7fb829e96640 1 -- 172.21.15.107:0/3403800785 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8240627b0 con 0x7fb82414be90 2024-03-25T19:41:53.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.078+0000 7fb80bfff640 1 -- 172.21.15.107:0/3403800785 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fb81c052490 con 0x7fb82414be90 2024-03-25T19:41:53.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.078+0000 7fb80bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:53.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.078+0000 7fb80bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:53.079 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:41:53.080 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:53.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 10 librados: watch_flush enter 2024-03-25T19:41:53.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 10 librados: watch_flush exit 2024-03-25T19:41:53.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-25T19:41:53.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 10 client.4486.objecter successfully canceled tick 2024-03-25T19:41:53.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 1 -- 172.21.15.107:0/3403800785 >> v1:172.21.15.195:6800/2984408421 conn(0x7fb81006dfb0 legacy=0x7fb810070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:53.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 10 monclient: shutdown 2024-03-25T19:41:53.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:53.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 1 -- 172.21.15.107:0/3403800785 >> v1:172.21.15.195:6789/0 conn(0x7fb82414be90 legacy=0x7fb8241e2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:53.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 1 -- 172.21.15.107:0/3403800785 shutdown_connections 2024-03-25T19:41:53.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 1 -- 172.21.15.107:0/3403800785 >> 172.21.15.107:0/3403800785 conn(0x7fb8240617a0 msgr2=0x7fb8240b63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:53.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 1 -- 172.21.15.107:0/3403800785 shutdown_connections 2024-03-25T19:41:53.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 1 -- 172.21.15.107:0/3403800785 wait complete. 2024-03-25T19:41:53.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:41:53.080+0000 7fb829e96640 1 librados: shutdown 2024-03-25T19:41:53.093 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:59.900 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:00.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.011+0000 7f86d6835640 10 monclient: get_monmap_and_config 2024-03-25T19:42:00.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.011+0000 7f86d6835640 10 monclient: build_initial_monmap 2024-03-25T19:42:00.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.011+0000 7f86d6835640 10 monclient: monmap: 2024-03-25T19:42:00.011 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:00.011 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:00.011 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:42:00.012337+0000 2024-03-25T19:42:00.011 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:42:00.012337+0000 2024-03-25T19:42:00.012 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:00.012 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:00.012 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:00.012 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:42:00.012 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:42:00.012 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:00.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.012+0000 7f86d6835640 1 Processor -- start 2024-03-25T19:42:00.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.012+0000 7f86d6835640 1 -- start start 2024-03-25T19:42:00.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.012+0000 7f86d6835640 10 monclient: init 2024-03-25T19:42:00.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.013+0000 7f86d6835640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:00.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.013+0000 7f86d6835640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:00.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.013+0000 7f86d6835640 10 monclient(hunting): picked mon.noname-c con 0x7f86d00a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:00.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.013+0000 7f86d6835640 10 monclient(hunting): picked mon.noname-b con 0x7f86d013f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:00.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.013+0000 7f86d6835640 10 monclient(hunting): picked mon.noname-a con 0x7f86d01430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:00.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.013+0000 7f86d6835640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d0142520 con 0x7f86d01430c0 2024-03-25T19:42:00.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.013+0000 7f86d6835640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d00b28a0 con 0x7f86d00a98f0 2024-03-25T19:42:00.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.013+0000 7f86d6835640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d00b2a30 con 0x7f86d013f710 2024-03-25T19:42:00.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.013+0000 7f86d6835640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:00.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.013+0000 7f86d6835640 10 monclient(hunting): authenticate will time out at 4912.354485s 2024-03-25T19:42:00.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.019+0000 7f86cf7fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f86d013f710 0x7f86d01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48108/0 (socket says 172.21.15.107:48108) 2024-03-25T19:42:00.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.019+0000 7f86cf7fe640 1 -- 172.21.15.107:0/1030397539 learned_addr learned my addr 172.21.15.107:0/1030397539 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:00.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.046+0000 7f86ceffd640 1 -- 172.21.15.107:0/1030397539 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596503240 0 0) 0x7f86d00b2a30 con 0x7f86d013f710 2024-03-25T19:42:00.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.046+0000 7f86ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.046+0000 7f86ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.046+0000 7f86ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.046+0000 7f86ceffd640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:42:00.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.046+0000 7f86ceffd640 1 -- 172.21.15.107:0/1030397539 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b8003520 con 0x7f86d013f710 2024-03-25T19:42:00.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 1 -- 172.21.15.107:0/1030397539 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348183716 0 0) 0x7f86b8003520 con 0x7f86d013f710 2024-03-25T19:42:00.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 1 -- 172.21.15.107:0/1030397539 >> v1:172.21.15.195:6789/0 conn(0x7f86d00a98f0 legacy=0x7f86d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 1 -- 172.21.15.107:0/1030397539 >> v1:172.21.15.107:6789/0 conn(0x7f86d01430c0 legacy=0x7f86d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:00.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 1 monclient: found mon.noname-b 2024-03-25T19:42:00.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:00.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:42:00.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 1 -- 172.21.15.107:0/1030397539 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d00b44c0 con 0x7f86d013f710 2024-03-25T19:42:00.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:00.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:00.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:00.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86d6835640 5 monclient: authenticate success, global_id 4502 2024-03-25T19:42:00.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86d6835640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:00.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 1 -- 172.21.15.107:0/1030397539 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f86c4001690 con 0x7f86d013f710 2024-03-25T19:42:00.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:00.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86ceffd640 10 monclient: dump: 2024-03-25T19:42:00.056 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:00.056 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:00.056 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.056 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.056 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.056 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:00.056 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:00.057 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:00.057 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:00.057 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:00.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.055+0000 7f86d6835640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:00.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.064+0000 7f86ceffd640 1 -- 172.21.15.107:0/1030397539 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86c40018e0 con 0x7f86d013f710 2024-03-25T19:42:00.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.064+0000 7f86ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:00.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.064+0000 7f86ceffd640 1 -- 172.21.15.107:0/1030397539 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f86c4001c70 con 0x7f86d013f710 2024-03-25T19:42:00.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.064+0000 7f86d6835640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:00.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.064+0000 7f86d6835640 4 set_mon_vals no callback set 2024-03-25T19:42:00.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.064+0000 7f86ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.064+0000 7f86ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:00.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.064+0000 7f86ceffd640 10 monclient: dump: 2024-03-25T19:42:00.064 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:00.064 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:00.064 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.065 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.065 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.065 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:00.065 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:00.065 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:00.065 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:00.065 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:00.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.065+0000 7f86d6835640 10 monclient: shutdown 2024-03-25T19:42:00.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.065+0000 7f86d6835640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:00.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.065+0000 7f86d6835640 1 -- 172.21.15.107:0/1030397539 >> v1:172.21.15.107:6790/0 conn(0x7f86d013f710 legacy=0x7f86d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.065+0000 7f86d6835640 1 -- 172.21.15.107:0/1030397539 shutdown_connections 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.065+0000 7f86d6835640 1 -- 172.21.15.107:0/1030397539 >> 172.21.15.107:0/1030397539 conn(0x7f86d00b7ea0 msgr2=0x7f86d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.065+0000 7f86d6835640 1 -- 172.21.15.107:0/1030397539 shutdown_connections 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.065+0000 7f86d6835640 1 -- 172.21.15.107:0/1030397539 wait complete. 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.066+0000 7f86d6835640 10 monclient: build_initial_monmap 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.066+0000 7f86d6835640 10 monclient: monmap: 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:00.067 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:42:00.068 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:42:00.068 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:00.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.066+0000 7f86d6835640 1 librados: starting msgr at 2024-03-25T19:42:00.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.066+0000 7f86d6835640 1 librados: starting objecter 2024-03-25T19:42:00.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.066+0000 7f86d6835640 1 Processor -- start 2024-03-25T19:42:00.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.067+0000 7f86d6835640 1 -- start start 2024-03-25T19:42:00.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.067+0000 7f86d6835640 1 librados: setting wanted keys 2024-03-25T19:42:00.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.067+0000 7f86d6835640 1 librados: calling monclient init 2024-03-25T19:42:00.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.067+0000 7f86d6835640 10 monclient: init 2024-03-25T19:42:00.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.067+0000 7f86d6835640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:00.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.067+0000 7f86d6835640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:00.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.067+0000 7f86d6835640 10 monclient(hunting): picked mon.noname-b con 0x7f86d00a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:00.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.068+0000 7f86d6835640 10 monclient(hunting): picked mon.noname-a con 0x7f86d013f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:00.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.068+0000 7f86d6835640 10 monclient(hunting): picked mon.noname-c con 0x7f86d01430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:00.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.068+0000 7f86d6835640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d00abb30 con 0x7f86d013f710 2024-03-25T19:42:00.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.068+0000 7f86d6835640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d01d8390 con 0x7f86d00a98f0 2024-03-25T19:42:00.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.068+0000 7f86d6835640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d0137f50 con 0x7f86d01430c0 2024-03-25T19:42:00.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.068+0000 7f86d6835640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:00.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.068+0000 7f86d6835640 10 monclient(hunting): authenticate will time out at 4912.408832s 2024-03-25T19:42:00.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.071+0000 7f86d4dd2640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f86d01430c0 0x7f86d0134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48142/0 (socket says 172.21.15.107:48142) 2024-03-25T19:42:00.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.071+0000 7f86d4dd2640 1 -- 172.21.15.107:0/3579936216 learned_addr learned my addr 172.21.15.107:0/3579936216 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:00.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.071+0000 7f86ccff9640 10 client.?.objecter ms_handle_connect 0x7f86d01430c0 2024-03-25T19:42:00.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.071+0000 7f86ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:00.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.076+0000 7f86ccff9640 10 client.?.objecter ms_handle_connect 0x7f86d00a98f0 2024-03-25T19:42:00.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.076+0000 7f86ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:00.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.076+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031368320 0 0) 0x7f86d0137f50 con 0x7f86d01430c0 2024-03-25T19:42:00.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.076+0000 7f86ccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.076+0000 7f86ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.076+0000 7f86ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.076+0000 7f86ccff9640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:42:00.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.076+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86bc004eb0 con 0x7f86d01430c0 2024-03-25T19:42:00.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.084+0000 7f86ccff9640 10 client.?.objecter ms_handle_connect 0x7f86d013f710 2024-03-25T19:42:00.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.084+0000 7f86ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:00.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.087+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273999832 0 0) 0x7f86d01d8390 con 0x7f86d00a98f0 2024-03-25T19:42:00.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.087+0000 7f86ccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.087+0000 7f86ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.087+0000 7f86ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.087+0000 7f86ccff9640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:42:00.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.087+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d0137f50 con 0x7f86d00a98f0 2024-03-25T19:42:00.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.096+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2760337130 0 0) 0x7f86bc004eb0 con 0x7f86d01430c0 2024-03-25T19:42:00.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.096+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d01d8390 con 0x7f86d01430c0 2024-03-25T19:42:00.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.096+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f86c8001c30 con 0x7f86d01430c0 2024-03-25T19:42:00.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.096+0000 7f86ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.096+0000 7f86ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:00.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.096+0000 7f86ccff9640 10 monclient(hunting): dump: 2024-03-25T19:42:00.096 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:00.096 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:00.097 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.097 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.097 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.097 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:00.097 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:00.097 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:00.097 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:00.097 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:00.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.096+0000 7f86ccff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:00.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.101+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765880802 0 0) 0x7f86d00abb30 con 0x7f86d013f710 2024-03-25T19:42:00.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.101+0000 7f86ccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.101+0000 7f86ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.101+0000 7f86ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.101+0000 7f86ccff9640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:42:00.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.101+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86bc004eb0 con 0x7f86d013f710 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.105+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2681553162 0 0) 0x7f86d0137f50 con 0x7f86d00a98f0 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.105+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d00abb30 con 0x7f86d00a98f0 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.105+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f86c00018a0 con 0x7f86d00a98f0 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.105+0000 7f86ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.105+0000 7f86ccff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.105+0000 7f86ccff9640 10 monclient(hunting): dump: 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:00.105 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:00.106 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:00.106 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:00.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.108+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3682656633 0 0) 0x7f86bc004eb0 con 0x7f86d013f710 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.108+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d0137f50 con 0x7f86d013f710 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.108+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f86c4001a50 con 0x7f86d013f710 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.108+0000 7f86ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.108+0000 7f86ccff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.108+0000 7f86ccff9640 10 monclient(hunting): dump: 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:00.108 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3754060123 0 0) 0x7f86d00abb30 con 0x7f86d00a98f0 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 >> v1:172.21.15.107:6790/0 conn(0x7f86d01430c0 legacy=0x7f86d0134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 >> v1:172.21.15.107:6789/0 conn(0x7f86d013f710 legacy=0x7f86d01d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86ccff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86ccff9640 1 monclient: found mon.b 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86ccff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0134720 con 0x7f86d00a98f0 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86ccff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86ccff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86d6835640 5 monclient: authenticate success, global_id 4492 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86d6835640 10 monclient: _renew_subs 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86d6835640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:00.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.114+0000 7f86d6835640 1 -- 172.21.15.107:0/3579936216 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d0134910 con 0x7f86d00a98f0 2024-03-25T19:42:00.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.115+0000 7f86d6835640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:00.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.115+0000 7f86d6835640 10 monclient: _renew_subs 2024-03-25T19:42:00.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.115+0000 7f86d6835640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:00.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.115+0000 7f86d6835640 1 -- 172.21.15.107:0/3579936216 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d0135050 con 0x7f86d00a98f0 2024-03-25T19:42:00.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.115+0000 7f86d6835640 1 librados: init done 2024-03-25T19:42:00.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.115+0000 7f86d6835640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:00.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.115+0000 7f86d6835640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:00.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.115+0000 7f86d6835640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:00.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.115+0000 7f86d6835640 1 -- 172.21.15.107:0/3579936216 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d00627b0 con 0x7f86d00a98f0 2024-03-25T19:42:00.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.392+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86c00035f0 con 0x7f86d00a98f0 2024-03-25T19:42:00.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.393+0000 7f86ccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:00.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.393+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f86c00039c0 con 0x7f86d00a98f0 2024-03-25T19:42:00.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.393+0000 7f86ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.393+0000 7f86ccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:00.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.393+0000 7f86ccff9640 10 monclient: dump: 2024-03-25T19:42:00.393 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:00.393 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:00.393 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.394 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:00.394 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.394 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:00.394 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:00.394 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:00.394 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:00.394 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:00.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.393+0000 7f86ce7fc640 4 set_mon_vals no callback set 2024-03-25T19:42:00.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.394+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f86c0003be0 con 0x7f86d00a98f0 2024-03-25T19:42:00.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.395+0000 7f86cf7fe640 10 monclient: get_auth_request con 0x7f86bc06e240 auth_method 0 2024-03-25T19:42:00.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.395+0000 7f86ccff9640 10 client.4492.objecter ms_handle_connect 0x7f86bc06e240 2024-03-25T19:42:00.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.398+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f86c0053570 con 0x7f86d00a98f0 2024-03-25T19:42:00.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.398+0000 7f86ccff9640 10 client.4492.objecter ms_dispatch 0x7f86d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:00.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.398+0000 7f86ccff9640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:00.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.398+0000 7f86ccff9640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:00.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.398+0000 7f86ccff9640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-25T19:42:00.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.399+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86c004f080 con 0x7f86d00a98f0 2024-03-25T19:42:00.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.399+0000 7f86ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:00.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.399+0000 7f86ccff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:00.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.631+0000 7f86d6835640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:00.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.631+0000 7f86d6835640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:00.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.631+0000 7f86d6835640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:00.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.631+0000 7f86d6835640 1 -- 172.21.15.107:0/3579936216 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86d013fdc0 con 0x7f86d00a98f0 2024-03-25T19:42:00.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.652+0000 7f86ccff9640 1 -- 172.21.15.107:0/3579936216 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f86c0050fb0 con 0x7f86d00a98f0 2024-03-25T19:42:00.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.652+0000 7f86ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:00.652 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.652+0000 7f86ccff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:00.652 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:42:00.652 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:00.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 10 librados: watch_flush enter 2024-03-25T19:42:00.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 10 librados: watch_flush exit 2024-03-25T19:42:00.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-25T19:42:00.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 10 client.4492.objecter successfully canceled tick 2024-03-25T19:42:00.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 1 -- 172.21.15.107:0/3579936216 >> v1:172.21.15.195:6800/2984408421 conn(0x7f86bc06e240 legacy=0x7f86bc070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 10 monclient: shutdown 2024-03-25T19:42:00.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:00.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 1 -- 172.21.15.107:0/3579936216 >> v1:172.21.15.195:6789/0 conn(0x7f86d00a98f0 legacy=0x7f86d0138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 1 -- 172.21.15.107:0/3579936216 shutdown_connections 2024-03-25T19:42:00.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 1 -- 172.21.15.107:0/3579936216 >> 172.21.15.107:0/3579936216 conn(0x7f86d00b7ea0 msgr2=0x7f86d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:00.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 1 -- 172.21.15.107:0/3579936216 shutdown_connections 2024-03-25T19:42:00.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.654+0000 7f86d6835640 1 -- 172.21.15.107:0/3579936216 wait complete. 2024-03-25T19:42:00.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:00.655+0000 7f86d6835640 1 librados: shutdown 2024-03-25T19:42:00.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:05.310 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:06:35.626 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.355 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:06:35.671 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.369 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:06:35.685 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.372 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:06:35.688 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.409 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:06:35.725 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.410 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:06:35.726 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.426 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:06:35.713 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.430 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:06:35.717 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.454 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:06:35.740 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:07.474 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:07.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.590+0000 7f4ee5b32640 10 monclient: get_monmap_and_config 2024-03-25T19:42:07.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.590+0000 7f4ee5b32640 10 monclient: build_initial_monmap 2024-03-25T19:42:07.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.590+0000 7f4ee5b32640 10 monclient: monmap: 2024-03-25T19:42:07.591 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:07.591 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:07.591 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:42:07.591778+0000 2024-03-25T19:42:07.591 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:42:07.591778+0000 2024-03-25T19:42:07.591 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:07.591 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:07.591 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:07.591 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:42:07.591 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:42:07.591 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:07.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.591+0000 7f4ee5b32640 1 Processor -- start 2024-03-25T19:42:07.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.591+0000 7f4ee5b32640 1 -- start start 2024-03-25T19:42:07.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.591+0000 7f4ee5b32640 10 monclient: init 2024-03-25T19:42:07.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.592+0000 7f4ee5b32640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:07.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.592+0000 7f4ee5b32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:07.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.592+0000 7f4ee5b32640 10 monclient(hunting): picked mon.noname-c con 0x7f4ee013fc60 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:07.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.592+0000 7f4ee5b32640 10 monclient(hunting): picked mon.noname-b con 0x7f4ee0070570 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:07.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.592+0000 7f4ee5b32640 10 monclient(hunting): picked mon.noname-a con 0x7f4ee00727a0 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:07.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.592+0000 7f4ee5b32640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee013c9f0 con 0x7f4ee00727a0 2024-03-25T19:42:07.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.592+0000 7f4ee5b32640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee006fac0 con 0x7f4ee013fc60 2024-03-25T19:42:07.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.592+0000 7f4ee5b32640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee00715d0 con 0x7f4ee0070570 2024-03-25T19:42:07.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.592+0000 7f4ee5b32640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:07.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.592+0000 7f4ee5b32640 10 monclient(hunting): authenticate will time out at 4919.933626s 2024-03-25T19:42:07.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.600+0000 7f4edeffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4ee0070570 0x7f4ee0072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43670/0 (socket says 172.21.15.107:43670) 2024-03-25T19:42:07.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.600+0000 7f4edeffd640 1 -- 172.21.15.107:0/1266718242 learned_addr learned my addr 172.21.15.107:0/1266718242 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:07.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.622+0000 7f4ede7fc640 1 -- 172.21.15.107:0/1266718242 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37053559 0 0) 0x7f4ee00715d0 con 0x7f4ee0070570 2024-03-25T19:42:07.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.622+0000 7f4ede7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.622+0000 7f4ede7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.622+0000 7f4ede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.622+0000 7f4ede7fc640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:42:07.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.622+0000 7f4ede7fc640 1 -- 172.21.15.107:0/1266718242 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec8005730 con 0x7f4ee0070570 2024-03-25T19:42:07.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.625+0000 7f4ede7fc640 1 -- 172.21.15.107:0/1266718242 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641157246 0 0) 0x7f4ee006fac0 con 0x7f4ee013fc60 2024-03-25T19:42:07.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.625+0000 7f4ede7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.625+0000 7f4ede7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.625+0000 7f4ede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.625+0000 7f4ede7fc640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:42:07.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.625+0000 7f4ede7fc640 1 -- 172.21.15.107:0/1266718242 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee00715d0 con 0x7f4ee013fc60 2024-03-25T19:42:07.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.630+0000 7f4ede7fc640 1 -- 172.21.15.107:0/1266718242 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2797046538 0 0) 0x7f4ec8005730 con 0x7f4ee0070570 2024-03-25T19:42:07.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.630+0000 7f4ede7fc640 1 -- 172.21.15.107:0/1266718242 >> v1:172.21.15.195:6789/0 conn(0x7f4ee013fc60 legacy=0x7f4ee006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.630+0000 7f4ede7fc640 1 -- 172.21.15.107:0/1266718242 >> v1:172.21.15.107:6789/0 conn(0x7f4ee00727a0 legacy=0x7f4ee014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.630+0000 7f4ede7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:07.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.630+0000 7f4ede7fc640 1 monclient: found mon.noname-b 2024-03-25T19:42:07.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.630+0000 7f4ede7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:07.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.630+0000 7f4ede7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:42:07.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.630+0000 7f4ede7fc640 1 -- 172.21.15.107:0/1266718242 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee00718b0 con 0x7f4ee0070570 2024-03-25T19:42:07.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.630+0000 7f4ede7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.630+0000 7f4ede7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.630+0000 7f4ede7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.631+0000 7f4ee5b32640 5 monclient: authenticate success, global_id 4508 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.631+0000 7f4ee5b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.631+0000 7f4ede7fc640 1 -- 172.21.15.107:0/1266718242 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4ed4001690 con 0x7f4ee0070570 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.631+0000 7f4ede7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.631+0000 7f4ede7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.631+0000 7f4ede7fc640 10 monclient: dump: 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:07.632 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:07.633 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:07.633 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:07.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.631+0000 7f4ee5b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:07.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.646+0000 7f4ede7fc640 1 -- 172.21.15.107:0/1266718242 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ed40018e0 con 0x7f4ee0070570 2024-03-25T19:42:07.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.646+0000 7f4ede7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.646+0000 7f4ede7fc640 1 -- 172.21.15.107:0/1266718242 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4ed4001c70 con 0x7f4ee0070570 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.646+0000 7f4ede7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.646+0000 7f4ede7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.646+0000 7f4ede7fc640 10 monclient: dump: 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:07.646 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:07.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.646+0000 7f4ee5b32640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:07.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.646+0000 7f4ee5b32640 4 set_mon_vals no callback set 2024-03-25T19:42:07.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.647+0000 7f4ee5b32640 10 monclient: shutdown 2024-03-25T19:42:07.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.647+0000 7f4ee5b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:07.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.647+0000 7f4ee5b32640 1 -- 172.21.15.107:0/1266718242 >> v1:172.21.15.107:6790/0 conn(0x7f4ee0070570 legacy=0x7f4ee0072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.647+0000 7f4ee5b32640 1 -- 172.21.15.107:0/1266718242 shutdown_connections 2024-03-25T19:42:07.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.647+0000 7f4ee5b32640 1 -- 172.21.15.107:0/1266718242 >> 172.21.15.107:0/1266718242 conn(0x7f4ee00b4a80 msgr2=0x7f4ee00b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:07.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.647+0000 7f4ee5b32640 1 -- 172.21.15.107:0/1266718242 shutdown_connections 2024-03-25T19:42:07.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.647+0000 7f4ee5b32640 1 -- 172.21.15.107:0/1266718242 wait complete. 2024-03-25T19:42:07.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.647+0000 7f4ee5b32640 10 monclient: build_initial_monmap 2024-03-25T19:42:07.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.647+0000 7f4ee5b32640 10 monclient: monmap: 2024-03-25T19:42:07.648 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:07.648 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.648+0000 7f4ee5b32640 1 librados: starting msgr at 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.648+0000 7f4ee5b32640 1 librados: starting objecter 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.648+0000 7f4ee5b32640 1 Processor -- start 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.648+0000 7f4ee5b32640 1 -- start start 2024-03-25T19:42:07.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.648+0000 7f4ee5b32640 1 librados: setting wanted keys 2024-03-25T19:42:07.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.648+0000 7f4ee5b32640 1 librados: calling monclient init 2024-03-25T19:42:07.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.648+0000 7f4ee5b32640 10 monclient: init 2024-03-25T19:42:07.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.649+0000 7f4ee5b32640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:07.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.649+0000 7f4ee5b32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:07.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.649+0000 7f4ee5b32640 10 monclient(hunting): picked mon.noname-a con 0x7f4ee0070570 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:07.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.649+0000 7f4ee5b32640 10 monclient(hunting): picked mon.noname-b con 0x7f4ee00727a0 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:07.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.649+0000 7f4ee5b32640 10 monclient(hunting): picked mon.noname-c con 0x7f4ee013fc60 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:07.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.649+0000 7f4ee5b32640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee01337e0 con 0x7f4ee0070570 2024-03-25T19:42:07.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.649+0000 7f4ee5b32640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee01417b0 con 0x7f4ee00727a0 2024-03-25T19:42:07.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.649+0000 7f4ee5b32640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee0139850 con 0x7f4ee013fc60 2024-03-25T19:42:07.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.649+0000 7f4ee5b32640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:07.652 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.649+0000 7f4ee5b32640 10 monclient(hunting): authenticate will time out at 4919.990169s 2024-03-25T19:42:07.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.655+0000 7f4edf7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4ee0070570 0x7f4ee01d6990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34524/0 (socket says 172.21.15.107:34524) 2024-03-25T19:42:07.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.655+0000 7f4edf7fe640 1 -- 172.21.15.107:0/812674526 learned_addr learned my addr 172.21.15.107:0/812674526 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:07.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.660+0000 7f4ec7fff640 10 client.?.objecter ms_handle_connect 0x7f4ee0070570 2024-03-25T19:42:07.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.661+0000 7f4ec7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:07.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.670+0000 7f4ec7fff640 10 client.?.objecter ms_handle_connect 0x7f4ee00727a0 2024-03-25T19:42:07.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.670+0000 7f4ec7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:07.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.674+0000 7f4ec7fff640 10 client.?.objecter ms_handle_connect 0x7f4ee013fc60 2024-03-25T19:42:07.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.674+0000 7f4ec7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:07.677 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.677+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820132330 0 0) 0x7f4ee01337e0 con 0x7f4ee0070570 2024-03-25T19:42:07.677 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.677+0000 7f4ec7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.677 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.677+0000 7f4ec7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.677 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.677+0000 7f4ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.677 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.677+0000 7f4ec7fff640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:42:07.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.678+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ecc004eb0 con 0x7f4ee0070570 2024-03-25T19:42:07.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.680+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408755470 0 0) 0x7f4ee01417b0 con 0x7f4ee00727a0 2024-03-25T19:42:07.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.680+0000 7f4ec7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.680+0000 7f4ec7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.680+0000 7f4ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.680+0000 7f4ec7fff640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:42:07.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.680+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee01337e0 con 0x7f4ee00727a0 2024-03-25T19:42:07.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.682+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909864341 0 0) 0x7f4ee0139850 con 0x7f4ee013fc60 2024-03-25T19:42:07.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.683+0000 7f4ec7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.683+0000 7f4ec7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.683+0000 7f4ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.683+0000 7f4ec7fff640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:42:07.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.683+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee01417b0 con 0x7f4ee013fc60 2024-03-25T19:42:07.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.701+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611243782 0 0) 0x7f4ee01337e0 con 0x7f4ee00727a0 2024-03-25T19:42:07.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.701+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ee0139850 con 0x7f4ee00727a0 2024-03-25T19:42:07.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.701+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4ed4001bf0 con 0x7f4ee00727a0 2024-03-25T19:42:07.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.701+0000 7f4ec7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.701+0000 7f4ec7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:07.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.701+0000 7f4ec7fff640 10 monclient(hunting): dump: 2024-03-25T19:42:07.701 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:07.701 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:07.701 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:07.702 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:07.702 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.702 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:07.702 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:07.702 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:07.702 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:07.702 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:07.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.701+0000 7f4ec7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:07.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.709+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086472277 0 0) 0x7f4ecc004eb0 con 0x7f4ee0070570 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.709+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ee01337e0 con 0x7f4ee0070570 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.709+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4ed0001990 con 0x7f4ee0070570 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.709+0000 7f4ec7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.709+0000 7f4ec7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.709+0000 7f4ec7fff640 10 monclient(hunting): dump: 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:07.709 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:07.710 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:07.710 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:07.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.716+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 878407703 0 0) 0x7f4ee01417b0 con 0x7f4ee013fc60 2024-03-25T19:42:07.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.716+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ecc004eb0 con 0x7f4ee013fc60 2024-03-25T19:42:07.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.716+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4ed8001b00 con 0x7f4ee013fc60 2024-03-25T19:42:07.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.716+0000 7f4ec7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.716+0000 7f4ec7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:07.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.716+0000 7f4ec7fff640 10 monclient(hunting): dump: 2024-03-25T19:42:07.716 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:07.716 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:07.716 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:07.717 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:07.717 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.717 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:07.717 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:07.717 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:07.717 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:07.717 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.718+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3544341795 0 0) 0x7f4ee0139850 con 0x7f4ee00727a0 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.718+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 >> v1:172.21.15.107:6790/0 conn(0x7f4ee013fc60 legacy=0x7f4ee01330d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 >> v1:172.21.15.107:6789/0 conn(0x7f4ee0070570 legacy=0x7f4ee01d6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ec7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ec7fff640 1 monclient: found mon.b 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ec7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ec7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee01432d0 con 0x7f4ee00727a0 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ec7fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ec7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ec7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:07.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ee5b32640 5 monclient: authenticate success, global_id 4498 2024-03-25T19:42:07.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ee5b32640 10 monclient: _renew_subs 2024-03-25T19:42:07.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ee5b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:07.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ee5b32640 1 -- 172.21.15.107:0/812674526 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee0140350 con 0x7f4ee00727a0 2024-03-25T19:42:07.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ee5b32640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:07.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ee5b32640 10 monclient: _renew_subs 2024-03-25T19:42:07.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ee5b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:07.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ee5b32640 1 -- 172.21.15.107:0/812674526 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee013a900 con 0x7f4ee00727a0 2024-03-25T19:42:07.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.719+0000 7f4ee5b32640 1 librados: init done 2024-03-25T19:42:07.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.720+0000 7f4ee5b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:07.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.720+0000 7f4ee5b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:07.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.720+0000 7f4ee5b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:07.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:07.720+0000 7f4ee5b32640 1 -- 172.21.15.107:0/812674526 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee01415f0 con 0x7f4ee00727a0 2024-03-25T19:42:08.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.043+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ed40010a0 con 0x7f4ee00727a0 2024-03-25T19:42:08.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.043+0000 7f4ec7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.043+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f4ed4004d90 con 0x7f4ee00727a0 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.043+0000 7f4ec7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.043+0000 7f4ec7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.043+0000 7f4ec7fff640 10 monclient: dump: 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:08.044 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:08.045 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:08.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.043+0000 7f4edd7fa640 4 set_mon_vals no callback set 2024-03-25T19:42:08.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.044+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f4ed401ad60 con 0x7f4ee00727a0 2024-03-25T19:42:08.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.044+0000 7f4edf7fe640 10 monclient: get_auth_request con 0x7f4ecc06e0b0 auth_method 0 2024-03-25T19:42:08.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.045+0000 7f4ec7fff640 10 client.4498.objecter ms_handle_connect 0x7f4ecc06e0b0 2024-03-25T19:42:08.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.049+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f4ed4054860 con 0x7f4ee00727a0 2024-03-25T19:42:08.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.049+0000 7f4ec7fff640 10 client.4498.objecter ms_dispatch 0x7f4ee0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:08.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.049+0000 7f4ec7fff640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:08.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.049+0000 7f4ec7fff640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:08.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.049+0000 7f4ec7fff640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T19:42:08.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.051+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ed4050370 con 0x7f4ee00727a0 2024-03-25T19:42:08.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.051+0000 7f4ec7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:08.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.051+0000 7f4ec7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:08.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.311+0000 7f4ee5b32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:08.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.311+0000 7f4ee5b32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:08.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.311+0000 7f4ee5b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:08.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.311+0000 7f4ee5b32640 1 -- 172.21.15.107:0/812674526 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ee00627b0 con 0x7f4ee00727a0 2024-03-25T19:42:08.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.336+0000 7f4ec7fff640 1 -- 172.21.15.107:0/812674526 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f4ed40522a0 con 0x7f4ee00727a0 2024-03-25T19:42:08.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.336+0000 7f4ec7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:08.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.336+0000 7f4ec7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:08.336 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:42:08.336 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:08.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 10 librados: watch_flush enter 2024-03-25T19:42:08.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 10 librados: watch_flush exit 2024-03-25T19:42:08.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T19:42:08.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 10 client.4498.objecter successfully canceled tick 2024-03-25T19:42:08.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 1 -- 172.21.15.107:0/812674526 >> v1:172.21.15.195:6800/2984408421 conn(0x7f4ecc06e0b0 legacy=0x7f4ecc070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:08.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 10 monclient: shutdown 2024-03-25T19:42:08.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:08.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 1 -- 172.21.15.107:0/812674526 >> v1:172.21.15.195:6789/0 conn(0x7f4ee00727a0 legacy=0x7f4ee01329c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:08.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 1 -- 172.21.15.107:0/812674526 shutdown_connections 2024-03-25T19:42:08.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 1 -- 172.21.15.107:0/812674526 >> 172.21.15.107:0/812674526 conn(0x7f4ee00b4a80 msgr2=0x7f4ee0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:08.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 1 -- 172.21.15.107:0/812674526 shutdown_connections 2024-03-25T19:42:08.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 1 -- 172.21.15.107:0/812674526 wait complete. 2024-03-25T19:42:08.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:08.338+0000 7f4ee5b32640 1 librados: shutdown 2024-03-25T19:42:08.351 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:15.158 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.273+0000 7f50d29e8640 10 monclient: get_monmap_and_config 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.273+0000 7f50d29e8640 10 monclient: build_initial_monmap 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.273+0000 7f50d29e8640 10 monclient: monmap: 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:42:15.274292+0000 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:42:15.274292+0000 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:42:15.273 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:15.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 1 Processor -- start 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 1 -- start start 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 10 monclient: init 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 10 monclient(hunting): picked mon.noname-b con 0x7f50cc0a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 10 monclient(hunting): picked mon.noname-a con 0x7f50cc13f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 10 monclient(hunting): picked mon.noname-c con 0x7f50cc1430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc142520 con 0x7f50cc13f710 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc0b28a0 con 0x7f50cc1430c0 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc0b2a30 con 0x7f50cc0a98f0 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:15.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.274+0000 7f50d29e8640 10 monclient(hunting): authenticate will time out at 4927.615715s 2024-03-25T19:42:15.281 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.281+0000 7f50cb7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f50cc13f710 0x7f50cc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59076/0 (socket says 172.21.15.107:59076) 2024-03-25T19:42:15.281 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.281+0000 7f50cb7fe640 1 -- 172.21.15.107:0/3909135299 learned_addr learned my addr 172.21.15.107:0/3909135299 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:15.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.292+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570638484 0 0) 0x7f50cc142520 con 0x7f50cc13f710 2024-03-25T19:42:15.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.292+0000 7f50caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:15.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.292+0000 7f50caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:15.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.292+0000 7f50caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:15.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.292+0000 7f50caffd640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:42:15.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.292+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b4003520 con 0x7f50cc13f710 2024-03-25T19:42:15.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.293+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010876228 0 0) 0x7f50cc0b28a0 con 0x7f50cc1430c0 2024-03-25T19:42:15.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.293+0000 7f50caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:15.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.293+0000 7f50caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:15.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.293+0000 7f50caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:15.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.293+0000 7f50caffd640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:42:15.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.293+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50cc142520 con 0x7f50cc1430c0 2024-03-25T19:42:15.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.306+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429713637 0 0) 0x7f50cc0b2a30 con 0x7f50cc0a98f0 2024-03-25T19:42:15.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.306+0000 7f50caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:15.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.306+0000 7f50caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:15.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.306+0000 7f50caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:15.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.306+0000 7f50caffd640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:42:15.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.306+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50cc0b28a0 con 0x7f50cc0a98f0 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.336+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2340022725 0 0) 0x7f50cc0b28a0 con 0x7f50cc0a98f0 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 >> v1:172.21.15.195:6789/0 conn(0x7f50cc1430c0 legacy=0x7f50cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 >> v1:172.21.15.107:6789/0 conn(0x7f50cc13f710 legacy=0x7f50cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 1 monclient: found mon.noname-b 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc0b44c0 con 0x7f50cc0a98f0 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f50bc0015b0 con 0x7f50cc0a98f0 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50d29e8640 5 monclient: authenticate success, global_id 4514 2024-03-25T19:42:15.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50d29e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50caffd640 10 monclient: dump: 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:15.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.337+0000 7f50d29e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:15.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.401+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50bc001710 con 0x7f50cc0a98f0 2024-03-25T19:42:15.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.401+0000 7f50caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:15.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.401+0000 7f50caffd640 1 -- 172.21.15.107:0/3909135299 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f50bc001b60 con 0x7f50cc0a98f0 2024-03-25T19:42:15.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.401+0000 7f50d29e8640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:15.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.401+0000 7f50d29e8640 4 set_mon_vals no callback set 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.401+0000 7f50caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.401+0000 7f50caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.401+0000 7f50caffd640 10 monclient: dump: 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:15.402 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:15.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.402+0000 7f50d29e8640 10 monclient: shutdown 2024-03-25T19:42:15.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.402+0000 7f50d29e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.402+0000 7f50d29e8640 1 -- 172.21.15.107:0/3909135299 >> v1:172.21.15.107:6790/0 conn(0x7f50cc0a98f0 legacy=0x7f50cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.402+0000 7f50d29e8640 1 -- 172.21.15.107:0/3909135299 shutdown_connections 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.402+0000 7f50d29e8640 1 -- 172.21.15.107:0/3909135299 >> 172.21.15.107:0/3909135299 conn(0x7f50cc0b7ea0 msgr2=0x7f50cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.402+0000 7f50d29e8640 1 -- 172.21.15.107:0/3909135299 shutdown_connections 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.402+0000 7f50d29e8640 1 -- 172.21.15.107:0/3909135299 wait complete. 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.403+0000 7f50d29e8640 10 monclient: build_initial_monmap 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.403+0000 7f50d29e8640 10 monclient: monmap: 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:15.404 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:42:15.405 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:42:15.405 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:15.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.403+0000 7f50d29e8640 1 librados: starting msgr at 2024-03-25T19:42:15.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.403+0000 7f50d29e8640 1 librados: starting objecter 2024-03-25T19:42:15.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.404+0000 7f50d29e8640 1 Processor -- start 2024-03-25T19:42:15.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.404+0000 7f50d29e8640 1 -- start start 2024-03-25T19:42:15.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.404+0000 7f50d29e8640 1 librados: setting wanted keys 2024-03-25T19:42:15.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.404+0000 7f50d29e8640 1 librados: calling monclient init 2024-03-25T19:42:15.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.404+0000 7f50d29e8640 10 monclient: init 2024-03-25T19:42:15.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.404+0000 7f50d29e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:15.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.405+0000 7f50d29e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:15.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.405+0000 7f50d29e8640 10 monclient(hunting): picked mon.noname-c con 0x7f50cc0a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:15.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.405+0000 7f50d29e8640 10 monclient(hunting): picked mon.noname-a con 0x7f50cc13f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:15.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.405+0000 7f50d29e8640 10 monclient(hunting): picked mon.noname-b con 0x7f50cc1430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:15.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.405+0000 7f50d29e8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc137f50 con 0x7f50cc13f710 2024-03-25T19:42:15.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.405+0000 7f50d29e8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc138150 con 0x7f50cc1430c0 2024-03-25T19:42:15.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.405+0000 7f50d29e8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc139970 con 0x7f50cc0a98f0 2024-03-25T19:42:15.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.405+0000 7f50d29e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:15.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.405+0000 7f50d29e8640 10 monclient(hunting): authenticate will time out at 4927.745968s 2024-03-25T19:42:15.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.409+0000 7f50cbfff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f50cc0a98f0 0x7f50cc138410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39862/0 (socket says 172.21.15.107:39862) 2024-03-25T19:42:15.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.409+0000 7f50cbfff640 1 -- 172.21.15.107:0/1751563461 learned_addr learned my addr 172.21.15.107:0/1751563461 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:15.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.411+0000 7f50c8ff9640 10 client.?.objecter ms_handle_connect 0x7f50cc1430c0 2024-03-25T19:42:15.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.411+0000 7f50c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:15.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.416+0000 7f50c8ff9640 10 client.?.objecter ms_handle_connect 0x7f50cc0a98f0 2024-03-25T19:42:15.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.416+0000 7f50c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:15.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.424+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398065873 0 0) 0x7f50cc139970 con 0x7f50cc0a98f0 2024-03-25T19:42:15.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.424+0000 7f50c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:15.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.424+0000 7f50c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:15.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.424+0000 7f50c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:15.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.424+0000 7f50c8ff9640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:42:15.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.424+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b8004eb0 con 0x7f50cc0a98f0 2024-03-25T19:42:15.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.427+0000 7f50c8ff9640 10 client.?.objecter ms_handle_connect 0x7f50cc13f710 2024-03-25T19:42:15.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.427+0000 7f50c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:15.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.430+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397523445 0 0) 0x7f50cc137f50 con 0x7f50cc13f710 2024-03-25T19:42:15.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.430+0000 7f50c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:15.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.430+0000 7f50c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:15.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.430+0000 7f50c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:15.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.430+0000 7f50c8ff9640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:42:15.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.430+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50cc139970 con 0x7f50cc13f710 2024-03-25T19:42:15.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.445+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 947096227 0 0) 0x7f50b8004eb0 con 0x7f50cc0a98f0 2024-03-25T19:42:15.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.445+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50cc137f50 con 0x7f50cc0a98f0 2024-03-25T19:42:15.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.446+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f50bc001b20 con 0x7f50cc0a98f0 2024-03-25T19:42:15.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.446+0000 7f50c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.446+0000 7f50c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.446+0000 7f50c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:15.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.446+0000 7f50c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:15.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.447+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 163771047 0 0) 0x7f50cc139970 con 0x7f50cc13f710 2024-03-25T19:42:15.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.447+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50b8004eb0 con 0x7f50cc13f710 2024-03-25T19:42:15.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.447+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f50c0001950 con 0x7f50cc13f710 2024-03-25T19:42:15.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.447+0000 7f50c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:15.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.447+0000 7f50c8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:15.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.447+0000 7f50c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:15.447 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:15.447 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:15.447 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:15.447 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:15.448 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:15.448 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:15.448 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:15.448 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:15.448 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:15.448 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3540202357 0 0) 0x7f50b8004eb0 con 0x7f50cc13f710 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 >> v1:172.21.15.107:6790/0 conn(0x7f50cc0a98f0 legacy=0x7f50cc138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 >> v1:172.21.15.195:6789/0 conn(0x7f50cc1430c0 legacy=0x7f50cc1e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50c8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50c8ff9640 1 monclient: found mon.a 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50c8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc1378d0 con 0x7f50cc13f710 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50c8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50c8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50d29e8640 5 monclient: authenticate success, global_id 4512 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50d29e8640 10 monclient: _renew_subs 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50d29e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50d29e8640 1 -- 172.21.15.107:0/1751563461 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50cc137ba0 con 0x7f50cc13f710 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50d29e8640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50d29e8640 10 monclient: _renew_subs 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50d29e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50d29e8640 1 -- 172.21.15.107:0/1751563461 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc1e34b0 con 0x7f50cc13f710 2024-03-25T19:42:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.464+0000 7f50d29e8640 1 librados: init done 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.465+0000 7f50d29e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.465+0000 7f50d29e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.465+0000 7f50d29e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.465+0000 7f50d29e8640 1 -- 172.21.15.107:0/1751563461 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc0627b0 con 0x7f50cc13f710 2024-03-25T19:42:15.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.764+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50c0001d60 con 0x7f50cc13f710 2024-03-25T19:42:15.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.764+0000 7f50c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:15.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.764+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f50c0004980 con 0x7f50cc13f710 2024-03-25T19:42:15.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.764+0000 7f50c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.764+0000 7f50c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.764+0000 7f50c8ff9640 10 monclient: dump: 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:15.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.764+0000 7f50ca7fc640 4 set_mon_vals no callback set 2024-03-25T19:42:15.766 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.765+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f50c0004ba0 con 0x7f50cc13f710 2024-03-25T19:42:15.766 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.766+0000 7f50cbfff640 10 monclient: get_auth_request con 0x7f50b806df10 auth_method 0 2024-03-25T19:42:15.766 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.766+0000 7f50c8ff9640 10 client.4512.objecter ms_handle_connect 0x7f50b806df10 2024-03-25T19:42:15.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.770+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f50c00546f0 con 0x7f50cc13f710 2024-03-25T19:42:15.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.770+0000 7f50c8ff9640 10 client.4512.objecter ms_dispatch 0x7f50cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:15.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.770+0000 7f50c8ff9640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:15.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.770+0000 7f50c8ff9640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:15.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.770+0000 7f50c8ff9640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-25T19:42:15.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.770+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50c0050200 con 0x7f50cc13f710 2024-03-25T19:42:15.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.770+0000 7f50c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:15.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:15.770+0000 7f50c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:16.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.015+0000 7f50d29e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:16.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.015+0000 7f50d29e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:16.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.015+0000 7f50d29e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:16.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.015+0000 7f50d29e8640 1 -- 172.21.15.107:0/1751563461 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50cc141c00 con 0x7f50cc13f710 2024-03-25T19:42:16.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.046+0000 7f50c8ff9640 1 -- 172.21.15.107:0/1751563461 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f50c0052130 con 0x7f50cc13f710 2024-03-25T19:42:16.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.047+0000 7f50c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:16.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.047+0000 7f50c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:16.047 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:42:16.047 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:16.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.049+0000 7f50d29e8640 10 librados: watch_flush enter 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.049+0000 7f50d29e8640 10 librados: watch_flush exit 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.049+0000 7f50d29e8640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.049+0000 7f50d29e8640 10 client.4512.objecter successfully canceled tick 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.049+0000 7f50d29e8640 1 -- 172.21.15.107:0/1751563461 >> v1:172.21.15.195:6800/2984408421 conn(0x7f50b806df10 legacy=0x7f50b80703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.050+0000 7f50d29e8640 10 monclient: shutdown 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.050+0000 7f50d29e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.050+0000 7f50d29e8640 1 -- 172.21.15.107:0/1751563461 >> v1:172.21.15.107:6789/0 conn(0x7f50cc13f710 legacy=0x7f50cc1db100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.050+0000 7f50d29e8640 1 -- 172.21.15.107:0/1751563461 shutdown_connections 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.050+0000 7f50d29e8640 1 -- 172.21.15.107:0/1751563461 >> 172.21.15.107:0/1751563461 conn(0x7f50cc0b7ea0 msgr2=0x7f50cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.050+0000 7f50d29e8640 1 -- 172.21.15.107:0/1751563461 shutdown_connections 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.050+0000 7f50d29e8640 1 -- 172.21.15.107:0/1751563461 wait complete. 2024-03-25T19:42:16.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:16.050+0000 7f50d29e8640 1 librados: shutdown 2024-03-25T19:42:16.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:22.867 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:22.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.985+0000 7f37a6740640 10 monclient: get_monmap_and_config 2024-03-25T19:42:22.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.985+0000 7f37a6740640 10 monclient: build_initial_monmap 2024-03-25T19:42:22.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.985+0000 7f37a6740640 10 monclient: monmap: 2024-03-25T19:42:22.985 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:22.985 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:22.985 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:42:22.986278+0000 2024-03-25T19:42:22.985 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:42:22.986278+0000 2024-03-25T19:42:22.985 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:22.985 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:22.985 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:22.985 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:42:22.986 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:42:22.986 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:22.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.986+0000 7f37a6740640 1 Processor -- start 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.986+0000 7f37a6740640 1 -- start start 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.986+0000 7f37a6740640 10 monclient: init 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.987+0000 7f37a6740640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.987+0000 7f37a6740640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.987+0000 7f37a6740640 10 monclient(hunting): picked mon.noname-b con 0x7f37a00b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.987+0000 7f37a6740640 10 monclient(hunting): picked mon.noname-c con 0x7f37a00b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.987+0000 7f37a6740640 10 monclient(hunting): picked mon.noname-a con 0x7f37a014be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.987+0000 7f37a6740640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a00af940 con 0x7f37a014be90 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.987+0000 7f37a6740640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a00b0a70 con 0x7f37a00b1730 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.987+0000 7f37a6740640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a00b0d50 con 0x7f37a00b4d60 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.987+0000 7f37a6740640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:22.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.987+0000 7f37a6740640 10 monclient(hunting): authenticate will time out at 4935.328198s 2024-03-25T19:42:22.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.996+0000 7f379ffff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f37a00b4d60 0x7f37a013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47328/0 (socket says 172.21.15.107:47328) 2024-03-25T19:42:22.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:22.996+0000 7f379ffff640 1 -- 172.21.15.107:0/1527672367 learned_addr learned my addr 172.21.15.107:0/1527672367 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:23.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.009+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750585062 0 0) 0x7f37a00b0a70 con 0x7f37a00b1730 2024-03-25T19:42:23.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.009+0000 7f379effd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:23.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.009+0000 7f379effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:23.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.009+0000 7f379effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:23.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.009+0000 7f379effd640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:42:23.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.009+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3788003520 con 0x7f37a00b1730 2024-03-25T19:42:23.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.019+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70395574 0 0) 0x7f37a00b0d50 con 0x7f37a00b4d60 2024-03-25T19:42:23.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.019+0000 7f379effd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:23.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.019+0000 7f379effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:23.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.019+0000 7f379effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:23.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.019+0000 7f379effd640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:42:23.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.019+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37a00b0a70 con 0x7f37a00b4d60 2024-03-25T19:42:23.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.021+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392048213 0 0) 0x7f37a00af940 con 0x7f37a014be90 2024-03-25T19:42:23.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.021+0000 7f379effd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:23.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.021+0000 7f379effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:23.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.021+0000 7f379effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:23.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.021+0000 7f379effd640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:42:23.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.021+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37a00b0d50 con 0x7f37a014be90 2024-03-25T19:42:23.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2022153642 0 0) 0x7f3788003520 con 0x7f37a00b1730 2024-03-25T19:42:23.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 >> v1:172.21.15.107:6790/0 conn(0x7f37a00b4d60 legacy=0x7f37a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 >> v1:172.21.15.107:6789/0 conn(0x7f37a014be90 legacy=0x7f37a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:23.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 1 monclient: found mon.noname-c 2024-03-25T19:42:23.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a00b1030 con 0x7f37a00b1730 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 10 monclient: _finish_auth 0 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3794001690 con 0x7f37a00b1730 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f37a6740640 5 monclient: authenticate success, global_id 4507 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f37a6740640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f379effd640 10 monclient: dump: 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:23.026 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:23.027 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:23.027 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:23.027 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:23.027 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:23.027 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:23.027 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:23.027 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:23.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.025+0000 7f37a6740640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:23.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.053+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37940017f0 con 0x7f37a00b1730 2024-03-25T19:42:23.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.053+0000 7f379effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:23.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.053+0000 7f379effd640 1 -- 172.21.15.107:0/1527672367 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3794001c00 con 0x7f37a00b1730 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.053+0000 7f379effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.053+0000 7f379effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.053+0000 7f379effd640 10 monclient: dump: 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.053+0000 7f37a6740640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:23.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.053+0000 7f37a6740640 4 set_mon_vals no callback set 2024-03-25T19:42:23.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.053+0000 7f37a6740640 10 monclient: shutdown 2024-03-25T19:42:23.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.053+0000 7f37a6740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:23.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.053+0000 7f37a6740640 1 -- 172.21.15.107:0/1527672367 >> v1:172.21.15.195:6789/0 conn(0x7f37a00b1730 legacy=0x7f37a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.054+0000 7f37a6740640 1 -- 172.21.15.107:0/1527672367 shutdown_connections 2024-03-25T19:42:23.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.054+0000 7f37a6740640 1 -- 172.21.15.107:0/1527672367 >> 172.21.15.107:0/1527672367 conn(0x7f37a00617a0 msgr2=0x7f37a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:23.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.054+0000 7f37a6740640 1 -- 172.21.15.107:0/1527672367 shutdown_connections 2024-03-25T19:42:23.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.054+0000 7f37a6740640 1 -- 172.21.15.107:0/1527672367 wait complete. 2024-03-25T19:42:23.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.054+0000 7f37a6740640 10 monclient: build_initial_monmap 2024-03-25T19:42:23.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.054+0000 7f37a6740640 10 monclient: monmap: 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.055+0000 7f37a6740640 1 librados: starting msgr at 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.055+0000 7f37a6740640 1 librados: starting objecter 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.055+0000 7f37a6740640 1 Processor -- start 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.055+0000 7f37a6740640 1 -- start start 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.055+0000 7f37a6740640 1 librados: setting wanted keys 2024-03-25T19:42:23.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.055+0000 7f37a6740640 1 librados: calling monclient init 2024-03-25T19:42:23.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.055+0000 7f37a6740640 10 monclient: init 2024-03-25T19:42:23.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.056+0000 7f37a6740640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:23.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.056+0000 7f37a6740640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:23.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.056+0000 7f37a6740640 10 monclient(hunting): picked mon.noname-c con 0x7f37a00b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:23.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.056+0000 7f37a6740640 10 monclient(hunting): picked mon.noname-a con 0x7f37a00b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:23.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.056+0000 7f37a6740640 10 monclient(hunting): picked mon.noname-b con 0x7f37a014be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:23.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.056+0000 7f37a6740640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a013bfb0 con 0x7f37a00b4d60 2024-03-25T19:42:23.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.056+0000 7f37a6740640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a01458e0 con 0x7f37a014be90 2024-03-25T19:42:23.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.056+0000 7f37a6740640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a012f7b0 con 0x7f37a00b1730 2024-03-25T19:42:23.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.056+0000 7f37a6740640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:23.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.056+0000 7f37a6740640 10 monclient(hunting): authenticate will time out at 4935.397547s 2024-03-25T19:42:23.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.072+0000 7f379f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f37a00b4d60 0x7f37a012e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46514/0 (socket says 172.21.15.107:46514) 2024-03-25T19:42:23.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.072+0000 7f379f7fe640 1 -- 172.21.15.107:0/638784990 learned_addr learned my addr 172.21.15.107:0/638784990 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:23.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.073+0000 7f379cff9640 10 client.?.objecter ms_handle_connect 0x7f37a00b4d60 2024-03-25T19:42:23.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.073+0000 7f379cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:23.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.079+0000 7f379cff9640 10 client.?.objecter ms_handle_connect 0x7f37a014be90 2024-03-25T19:42:23.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.079+0000 7f379cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:23.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.086+0000 7f379cff9640 10 client.?.objecter ms_handle_connect 0x7f37a00b1730 2024-03-25T19:42:23.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.086+0000 7f379cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:23.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.087+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047511190 0 0) 0x7f37a013bfb0 con 0x7f37a00b4d60 2024-03-25T19:42:23.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.087+0000 7f379cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:23.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.087+0000 7f379cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:23.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.087+0000 7f379cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:23.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.087+0000 7f379cff9640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:42:23.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.087+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f378c004eb0 con 0x7f37a00b4d60 2024-03-25T19:42:23.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.089+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037197644 0 0) 0x7f37a01458e0 con 0x7f37a014be90 2024-03-25T19:42:23.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.089+0000 7f379cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:23.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.089+0000 7f379cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:23.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.089+0000 7f379cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:23.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.089+0000 7f379cff9640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:42:23.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.089+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37a013bfb0 con 0x7f37a014be90 2024-03-25T19:42:23.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.099+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3118898020 0 0) 0x7f378c004eb0 con 0x7f37a00b4d60 2024-03-25T19:42:23.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.099+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37a01458e0 con 0x7f37a00b4d60 2024-03-25T19:42:23.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.103+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013017242 0 0) 0x7f37a012f7b0 con 0x7f37a00b1730 2024-03-25T19:42:23.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.103+0000 7f379cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:23.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.103+0000 7f379cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:23.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.103+0000 7f379cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:23.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.103+0000 7f379cff9640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:42:23.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.103+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f378c004eb0 con 0x7f37a00b1730 2024-03-25T19:42:23.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.104+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3794001c40 con 0x7f37a00b4d60 2024-03-25T19:42:23.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.104+0000 7f379cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.104+0000 7f379cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.104+0000 7f379cff9640 10 monclient(hunting): dump: 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:23.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.104+0000 7f379cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:23.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.107+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 910664433 0 0) 0x7f37a01458e0 con 0x7f37a00b4d60 2024-03-25T19:42:23.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 >> v1:172.21.15.107:6790/0 conn(0x7f37a00b1730 legacy=0x7f37a013b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 >> v1:172.21.15.195:6789/0 conn(0x7f37a014be90 legacy=0x7f37a012f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f379cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:23.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f379cff9640 1 monclient: found mon.a 2024-03-25T19:42:23.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f379cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:23.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f379cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:23.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a013aa90 con 0x7f37a00b4d60 2024-03-25T19:42:23.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f379cff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:23.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f379cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:23.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f379cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:23.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f37a6740640 5 monclient: authenticate success, global_id 4518 2024-03-25T19:42:23.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f37a6740640 10 monclient: _renew_subs 2024-03-25T19:42:23.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f37a6740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:23.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f37a6740640 1 -- 172.21.15.107:0/638784990 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a013ad70 con 0x7f37a00b4d60 2024-03-25T19:42:23.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f37a6740640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:23.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f37a6740640 10 monclient: _renew_subs 2024-03-25T19:42:23.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f37a6740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:23.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f37a6740640 1 -- 172.21.15.107:0/638784990 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a0145330 con 0x7f37a00b4d60 2024-03-25T19:42:23.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.108+0000 7f37a6740640 1 librados: init done 2024-03-25T19:42:23.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.109+0000 7f37a6740640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:23.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.109+0000 7f37a6740640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:23.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.109+0000 7f37a6740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:23.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.109+0000 7f37a6740640 1 -- 172.21.15.107:0/638784990 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a01401f0 con 0x7f37a00b4d60 2024-03-25T19:42:23.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.413+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37940011f0 con 0x7f37a00b4d60 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.413+0000 7f379cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.413+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3794003a80 con 0x7f37a00b4d60 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.413+0000 7f379cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.413+0000 7f379cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.413+0000 7f379cff9640 10 monclient: dump: 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:23.413 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:23.414 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:23.414 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:23.414 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:23.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.413+0000 7f379e7fc640 4 set_mon_vals no callback set 2024-03-25T19:42:23.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.413+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f3794003ca0 con 0x7f37a00b4d60 2024-03-25T19:42:23.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.414+0000 7f379ffff640 10 monclient: get_auth_request con 0x7f378c06ddb0 auth_method 0 2024-03-25T19:42:23.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.414+0000 7f379cff9640 10 client.4518.objecter ms_handle_connect 0x7f378c06ddb0 2024-03-25T19:42:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.420+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f3794054990 con 0x7f37a00b4d60 2024-03-25T19:42:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.420+0000 7f379cff9640 10 client.4518.objecter ms_dispatch 0x7f37a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.420+0000 7f379cff9640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.420+0000 7f379cff9640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.420+0000 7f379cff9640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-25T19:42:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.420+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37940502d0 con 0x7f37a00b4d60 2024-03-25T19:42:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.420+0000 7f379cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.420+0000 7f379cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:23.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.643+0000 7f37a6740640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:23.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.644+0000 7f37a6740640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:23.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.644+0000 7f37a6740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:23.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.644+0000 7f37a6740640 1 -- 172.21.15.107:0/638784990 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37a00afbd0 con 0x7f37a00b4d60 2024-03-25T19:42:23.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.675+0000 7f379cff9640 1 -- 172.21.15.107:0/638784990 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f37940523d0 con 0x7f37a00b4d60 2024-03-25T19:42:23.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.676+0000 7f379cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:23.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.676+0000 7f379cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:23.676 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:42:23.676 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:23.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 10 librados: watch_flush enter 2024-03-25T19:42:23.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 10 librados: watch_flush exit 2024-03-25T19:42:23.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-25T19:42:23.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 10 client.4518.objecter successfully canceled tick 2024-03-25T19:42:23.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 1 -- 172.21.15.107:0/638784990 >> v1:172.21.15.195:6800/2984408421 conn(0x7f378c06ddb0 legacy=0x7f378c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 10 monclient: shutdown 2024-03-25T19:42:23.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:23.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 1 -- 172.21.15.107:0/638784990 >> v1:172.21.15.107:6789/0 conn(0x7f37a00b4d60 legacy=0x7f37a012e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:23.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 1 -- 172.21.15.107:0/638784990 shutdown_connections 2024-03-25T19:42:23.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 1 -- 172.21.15.107:0/638784990 >> 172.21.15.107:0/638784990 conn(0x7f37a00617a0 msgr2=0x7f37a013c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:23.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 1 -- 172.21.15.107:0/638784990 shutdown_connections 2024-03-25T19:42:23.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 1 -- 172.21.15.107:0/638784990 wait complete. 2024-03-25T19:42:23.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:23.678+0000 7f37a6740640 1 librados: shutdown 2024-03-25T19:42:23.690 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:25.416 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:06:55.732 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:25.423 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:06:55.739 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:25.465 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:06:55.781 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:25.470 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:06:55.756 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:25.475 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:06:55.791 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:25.484 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:06:55.771 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:25.488 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:06:55.804 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:25.511 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:06:55.797 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:25.518 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:06:55.834 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:30.496 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:30.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.594+0000 7f59481d6640 10 monclient: get_monmap_and_config 2024-03-25T19:42:30.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.594+0000 7f59481d6640 10 monclient: build_initial_monmap 2024-03-25T19:42:30.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.594+0000 7f59481d6640 10 monclient: monmap: 2024-03-25T19:42:30.594 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:30.594 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:30.594 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:42:30.595652+0000 2024-03-25T19:42:30.594 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:42:30.595652+0000 2024-03-25T19:42:30.594 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:30.595 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:30.595 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:30.595 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:42:30.595 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:42:30.595 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:30.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.595+0000 7f59481d6640 1 Processor -- start 2024-03-25T19:42:30.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.595+0000 7f59481d6640 1 -- start start 2024-03-25T19:42:30.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.595+0000 7f59481d6640 10 monclient: init 2024-03-25T19:42:30.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.596+0000 7f59481d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:30.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.596+0000 7f59481d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:30.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.596+0000 7f59481d6640 10 monclient(hunting): picked mon.noname-b con 0x7f59400b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:30.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.596+0000 7f59481d6640 10 monclient(hunting): picked mon.noname-c con 0x7f59400b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:30.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.596+0000 7f59481d6640 10 monclient(hunting): picked mon.noname-a con 0x7f594014be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:30.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.596+0000 7f59481d6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59400af940 con 0x7f594014be90 2024-03-25T19:42:30.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.596+0000 7f59481d6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59400b0a70 con 0x7f59400b1730 2024-03-25T19:42:30.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.596+0000 7f59481d6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59400b0d50 con 0x7f59400b4d60 2024-03-25T19:42:30.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.596+0000 7f59481d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:30.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.596+0000 7f59481d6640 10 monclient(hunting): authenticate will time out at 4942.937014s 2024-03-25T19:42:30.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.600+0000 7f5945f72640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f59400b4d60 0x7f594013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50984/0 (socket says 172.21.15.107:50984) 2024-03-25T19:42:30.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.600+0000 7f5945f72640 1 -- 172.21.15.107:0/1771610473 learned_addr learned my addr 172.21.15.107:0/1771610473 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:30.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.612+0000 7f5944f70640 1 -- 172.21.15.107:0/1771610473 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482287289 0 0) 0x7f59400b0d50 con 0x7f59400b4d60 2024-03-25T19:42:30.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.613+0000 7f5944f70640 10 monclient(hunting): _negotiate 2024-03-25T19:42:30.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.613+0000 7f5944f70640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:30.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.613+0000 7f5944f70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:30.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.613+0000 7f5944f70640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:42:30.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.613+0000 7f5944f70640 1 -- 172.21.15.107:0/1771610473 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f592c003520 con 0x7f59400b4d60 2024-03-25T19:42:30.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 1 -- 172.21.15.107:0/1771610473 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 742679746 0 0) 0x7f592c003520 con 0x7f59400b4d60 2024-03-25T19:42:30.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 1 -- 172.21.15.107:0/1771610473 >> v1:172.21.15.195:6789/0 conn(0x7f59400b1730 legacy=0x7f59400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 1 -- 172.21.15.107:0/1771610473 >> v1:172.21.15.107:6789/0 conn(0x7f594014be90 legacy=0x7f594014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 10 monclient: _finish_hunting 0 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 1 monclient: found mon.noname-b 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 1 -- 172.21.15.107:0/1771610473 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59400b1030 con 0x7f59400b4d60 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 10 monclient: _finish_auth 0 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 10 monclient: _check_auth_tickets 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 1 -- 172.21.15.107:0/1771610473 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f593c001690 con 0x7f59400b4d60 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.632+0000 7f5944f70640 10 monclient: dump: 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:30.633 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:30.634 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:30.634 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:30.634 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:30.634 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:30.634 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:30.634 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:30.634 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:30.634 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:30.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.633+0000 7f59481d6640 5 monclient: authenticate success, global_id 4526 2024-03-25T19:42:30.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.633+0000 7f59481d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:30.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.699+0000 7f5944f70640 1 -- 172.21.15.107:0/1771610473 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f593c0017f0 con 0x7f59400b4d60 2024-03-25T19:42:30.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.699+0000 7f5944f70640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:30.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.699+0000 7f5944f70640 1 -- 172.21.15.107:0/1771610473 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f593c001c40 con 0x7f59400b4d60 2024-03-25T19:42:30.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.699+0000 7f59481d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:30.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.699+0000 7f59481d6640 4 set_mon_vals no callback set 2024-03-25T19:42:30.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.699+0000 7f5944f70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.699+0000 7f5944f70640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.699+0000 7f5944f70640 10 monclient: dump: 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:30.700 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:30.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.700+0000 7f59481d6640 10 monclient: shutdown 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.700+0000 7f59481d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.700+0000 7f59481d6640 1 -- 172.21.15.107:0/1771610473 >> v1:172.21.15.107:6790/0 conn(0x7f59400b4d60 legacy=0x7f594013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.700+0000 7f59481d6640 1 -- 172.21.15.107:0/1771610473 shutdown_connections 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.700+0000 7f59481d6640 1 -- 172.21.15.107:0/1771610473 >> 172.21.15.107:0/1771610473 conn(0x7f59400617a0 msgr2=0x7f5940061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.700+0000 7f59481d6640 1 -- 172.21.15.107:0/1771610473 shutdown_connections 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.700+0000 7f59481d6640 1 -- 172.21.15.107:0/1771610473 wait complete. 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.701+0000 7f59481d6640 10 monclient: build_initial_monmap 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.701+0000 7f59481d6640 10 monclient: monmap: 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:42:30.702 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.701+0000 7f59481d6640 1 librados: starting msgr at 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.701+0000 7f59481d6640 1 librados: starting objecter 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.701+0000 7f59481d6640 1 Processor -- start 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.701+0000 7f59481d6640 1 -- start start 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 1 librados: setting wanted keys 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 1 librados: calling monclient init 2024-03-25T19:42:30.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 10 monclient: init 2024-03-25T19:42:30.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:30.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:30.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 10 monclient(hunting): picked mon.noname-c con 0x7f59400b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:30.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 10 monclient(hunting): picked mon.noname-a con 0x7f59400b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:30.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 10 monclient(hunting): picked mon.noname-b con 0x7f594014be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:30.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59401454c0 con 0x7f59400b4d60 2024-03-25T19:42:30.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59401457a0 con 0x7f594014be90 2024-03-25T19:42:30.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5940140380 con 0x7f59400b1730 2024-03-25T19:42:30.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:30.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.702+0000 7f59481d6640 10 monclient(hunting): authenticate will time out at 4943.043624s 2024-03-25T19:42:30.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.712+0000 7f5946773640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f594014be90 0x7f59401e2270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:55836/0 (socket says 172.21.15.107:55836) 2024-03-25T19:42:30.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.712+0000 7f5946773640 1 -- 172.21.15.107:0/3745317942 learned_addr learned my addr 172.21.15.107:0/3745317942 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:30.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.716+0000 7f59367fc640 10 client.?.objecter ms_handle_connect 0x7f594014be90 2024-03-25T19:42:30.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.716+0000 7f59367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:30.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.718+0000 7f59367fc640 10 client.?.objecter ms_handle_connect 0x7f59400b4d60 2024-03-25T19:42:30.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.718+0000 7f59367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:30.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.728+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648781234 0 0) 0x7f59401457a0 con 0x7f594014be90 2024-03-25T19:42:30.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.728+0000 7f59367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:30.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.728+0000 7f59367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:30.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.728+0000 7f59367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:30.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.728+0000 7f59367fc640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:42:30.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.728+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5924004eb0 con 0x7f594014be90 2024-03-25T19:42:30.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.732+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169257821 0 0) 0x7f59401454c0 con 0x7f59400b4d60 2024-03-25T19:42:30.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.732+0000 7f59367fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:30.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.732+0000 7f59367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:30.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.732+0000 7f59367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:30.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.732+0000 7f59367fc640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:42:30.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.732+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59401457a0 con 0x7f59400b4d60 2024-03-25T19:42:30.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.739+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3356675043 0 0) 0x7f59401457a0 con 0x7f59400b4d60 2024-03-25T19:42:30.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.739+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59401454c0 con 0x7f59400b4d60 2024-03-25T19:42:30.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.739+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5930001930 con 0x7f59400b4d60 2024-03-25T19:42:30.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.739+0000 7f59367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:30.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.739+0000 7f59367fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:30.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.739+0000 7f59367fc640 10 monclient(hunting): dump: 2024-03-25T19:42:30.739 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:30.740 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:30.740 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:30.740 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:30.740 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:30.740 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:30.740 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:30.740 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:30.740 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:30.740 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:30.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.739+0000 7f59367fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:30.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.741+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1211868441 0 0) 0x7f5924004eb0 con 0x7f594014be90 2024-03-25T19:42:30.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.741+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59401457a0 con 0x7f594014be90 2024-03-25T19:42:30.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.741+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5938001c30 con 0x7f594014be90 2024-03-25T19:42:30.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.741+0000 7f59367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:30.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.741+0000 7f59367fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:30.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.741+0000 7f59367fc640 10 monclient(hunting): dump: 2024-03-25T19:42:30.741 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:30.741 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:30.741 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:30.741 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:30.742 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:30.742 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:30.742 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:30.742 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:30.742 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:30.742 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:30.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3179058015 0 0) 0x7f59401454c0 con 0x7f59400b4d60 2024-03-25T19:42:30.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 >> v1:172.21.15.107:6790/0 conn(0x7f59400b1730 legacy=0x7f5940140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 >> v1:172.21.15.195:6789/0 conn(0x7f594014be90 legacy=0x7f59401e2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59367fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:30.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59367fc640 1 monclient: found mon.a 2024-03-25T19:42:30.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:30.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59367fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:30.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59401447b0 con 0x7f59400b4d60 2024-03-25T19:42:30.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59367fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:30.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59367fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:30.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:30.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59481d6640 5 monclient: authenticate success, global_id 4521 2024-03-25T19:42:30.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59481d6640 10 monclient: _renew_subs 2024-03-25T19:42:30.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59481d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:30.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59481d6640 1 -- 172.21.15.107:0/3745317942 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5940144a60 con 0x7f59400b4d60 2024-03-25T19:42:30.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59481d6640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:30.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59481d6640 10 monclient: _renew_subs 2024-03-25T19:42:30.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59481d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:30.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59481d6640 1 -- 172.21.15.107:0/3745317942 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f594013e820 con 0x7f59400b4d60 2024-03-25T19:42:30.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.744+0000 7f59481d6640 1 librados: init done 2024-03-25T19:42:30.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.745+0000 7f59481d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:30.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.745+0000 7f59481d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:30.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.745+0000 7f59481d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:30.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:30.745+0000 7f59481d6640 1 -- 172.21.15.107:0/3745317942 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59401401f0 con 0x7f59400b4d60 2024-03-25T19:42:31.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.054+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5930003610 con 0x7f59400b4d60 2024-03-25T19:42:31.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.054+0000 7f59367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:31.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.054+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5930003960 con 0x7f59400b4d60 2024-03-25T19:42:31.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.054+0000 7f59367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.054+0000 7f59367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.054+0000 7f59367fc640 10 monclient: dump: 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:31.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.054+0000 7f5937fff640 4 set_mon_vals no callback set 2024-03-25T19:42:31.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.055+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f5930003b80 con 0x7f59400b4d60 2024-03-25T19:42:31.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.056+0000 7f5945f72640 10 monclient: get_auth_request con 0x7f592406de30 auth_method 0 2024-03-25T19:42:31.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.056+0000 7f59367fc640 10 client.4521.objecter ms_handle_connect 0x7f592406de30 2024-03-25T19:42:31.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.061+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f59300535e0 con 0x7f59400b4d60 2024-03-25T19:42:31.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.061+0000 7f59367fc640 10 client.4521.objecter ms_dispatch 0x7f5940000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:31.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.061+0000 7f59367fc640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:31.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.061+0000 7f59367fc640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:31.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.061+0000 7f59367fc640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T19:42:31.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.061+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f593004f0f0 con 0x7f59400b4d60 2024-03-25T19:42:31.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.061+0000 7f59367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:31.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.061+0000 7f59367fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:31.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.315+0000 7f59481d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:31.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.315+0000 7f59481d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:31.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.315+0000 7f59481d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:31.315 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.315+0000 7f59481d6640 1 -- 172.21.15.107:0/3745317942 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5940140510 con 0x7f59400b4d60 2024-03-25T19:42:31.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.343+0000 7f59367fc640 1 -- 172.21.15.107:0/3745317942 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f5930051020 con 0x7f59400b4d60 2024-03-25T19:42:31.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.343+0000 7f59367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:31.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.343+0000 7f59367fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:31.343 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:42:31.343 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:31.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 10 librados: watch_flush enter 2024-03-25T19:42:31.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 10 librados: watch_flush exit 2024-03-25T19:42:31.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T19:42:31.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 10 client.4521.objecter successfully canceled tick 2024-03-25T19:42:31.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 1 -- 172.21.15.107:0/3745317942 >> v1:172.21.15.195:6800/2984408421 conn(0x7f592406de30 legacy=0x7f59240702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:31.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 10 monclient: shutdown 2024-03-25T19:42:31.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:31.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 1 -- 172.21.15.107:0/3745317942 >> v1:172.21.15.107:6789/0 conn(0x7f59400b4d60 legacy=0x7f5940143a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:31.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 1 -- 172.21.15.107:0/3745317942 shutdown_connections 2024-03-25T19:42:31.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 1 -- 172.21.15.107:0/3745317942 >> 172.21.15.107:0/3745317942 conn(0x7f59400617a0 msgr2=0x7f5940137dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:31.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 1 -- 172.21.15.107:0/3745317942 shutdown_connections 2024-03-25T19:42:31.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 1 -- 172.21.15.107:0/3745317942 wait complete. 2024-03-25T19:42:31.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:31.346+0000 7f59481d6640 1 librados: shutdown 2024-03-25T19:42:31.358 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:38.164 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:38.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.276+0000 7f57aae7b640 10 monclient: get_monmap_and_config 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.276+0000 7f57aae7b640 10 monclient: build_initial_monmap 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.276+0000 7f57aae7b640 10 monclient: monmap: 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:42:38.277058+0000 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:42:38.277058+0000 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:42:38.276 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:38.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.276+0000 7f57aae7b640 1 Processor -- start 2024-03-25T19:42:38.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 1 -- start start 2024-03-25T19:42:38.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 10 monclient: init 2024-03-25T19:42:38.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:38.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:38.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 10 monclient(hunting): picked mon.noname-c con 0x7f57a40b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:38.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 10 monclient(hunting): picked mon.noname-b con 0x7f57a40b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:38.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 10 monclient(hunting): picked mon.noname-a con 0x7f57a414be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:38.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a40af940 con 0x7f57a414be90 2024-03-25T19:42:38.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a40b0a70 con 0x7f57a40b4d60 2024-03-25T19:42:38.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a40b0d50 con 0x7f57a40b1730 2024-03-25T19:42:38.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:38.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.277+0000 7f57aae7b640 10 monclient(hunting): authenticate will time out at 4950.618452s 2024-03-25T19:42:38.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.278+0000 7f57a8c17640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f57a40b4d60 0x7f57a413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:60802/0 (socket says 172.21.15.107:60802) 2024-03-25T19:42:38.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.278+0000 7f57a8c17640 1 -- 172.21.15.107:0/1389908810 learned_addr learned my addr 172.21.15.107:0/1389908810 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:38.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.343+0000 7f57a37fe640 1 -- 172.21.15.107:0/1389908810 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572582214 0 0) 0x7f57a40b0d50 con 0x7f57a40b1730 2024-03-25T19:42:38.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.343+0000 7f57a37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:38.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.343+0000 7f57a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:38.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.343+0000 7f57a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:38.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.343+0000 7f57a37fe640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:42:38.343 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.343+0000 7f57a37fe640 1 -- 172.21.15.107:0/1389908810 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5794003520 con 0x7f57a40b1730 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 1 -- 172.21.15.107:0/1389908810 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2179880104 0 0) 0x7f5794003520 con 0x7f57a40b1730 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 1 -- 172.21.15.107:0/1389908810 >> v1:172.21.15.195:6789/0 conn(0x7f57a40b4d60 legacy=0x7f57a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 1 -- 172.21.15.107:0/1389908810 >> v1:172.21.15.107:6789/0 conn(0x7f57a414be90 legacy=0x7f57a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 1 monclient: found mon.noname-b 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 1 -- 172.21.15.107:0/1389908810 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a40b1030 con 0x7f57a40b1730 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:38.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 1 -- 172.21.15.107:0/1389908810 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5790001690 con 0x7f57a40b1730 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57a37fe640 10 monclient: dump: 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57aae7b640 5 monclient: authenticate success, global_id 4529 2024-03-25T19:42:38.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.364+0000 7f57aae7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:38.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.379+0000 7f57a37fe640 1 -- 172.21.15.107:0/1389908810 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57900017f0 con 0x7f57a40b1730 2024-03-25T19:42:38.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.379+0000 7f57a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:38.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.379+0000 7f57a37fe640 1 -- 172.21.15.107:0/1389908810 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5790001c00 con 0x7f57a40b1730 2024-03-25T19:42:38.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.379+0000 7f57aae7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:38.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.379+0000 7f57aae7b640 4 set_mon_vals no callback set 2024-03-25T19:42:38.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.379+0000 7f57a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:38.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.379+0000 7f57a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:38.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.379+0000 7f57a37fe640 10 monclient: dump: 2024-03-25T19:42:38.379 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:38.379 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:38.379 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:38.380 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:38.380 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:38.380 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:38.380 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:38.380 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:38.380 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:38.380 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:38.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.380+0000 7f57aae7b640 10 monclient: shutdown 2024-03-25T19:42:38.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.380+0000 7f57aae7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:38.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.380+0000 7f57aae7b640 1 -- 172.21.15.107:0/1389908810 >> v1:172.21.15.107:6790/0 conn(0x7f57a40b1730 legacy=0x7f57a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.380+0000 7f57aae7b640 1 -- 172.21.15.107:0/1389908810 shutdown_connections 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.380+0000 7f57aae7b640 1 -- 172.21.15.107:0/1389908810 >> 172.21.15.107:0/1389908810 conn(0x7f57a40617a0 msgr2=0x7f57a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.380+0000 7f57aae7b640 1 -- 172.21.15.107:0/1389908810 shutdown_connections 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.380+0000 7f57aae7b640 1 -- 172.21.15.107:0/1389908810 wait complete. 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.381+0000 7f57aae7b640 10 monclient: build_initial_monmap 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.381+0000 7f57aae7b640 10 monclient: monmap: 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:42:38.382 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:42:38.383 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:38.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.381+0000 7f57aae7b640 1 librados: starting msgr at 2024-03-25T19:42:38.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.381+0000 7f57aae7b640 1 librados: starting objecter 2024-03-25T19:42:38.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.381+0000 7f57aae7b640 1 Processor -- start 2024-03-25T19:42:38.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.382+0000 7f57aae7b640 1 -- start start 2024-03-25T19:42:38.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.382+0000 7f57aae7b640 1 librados: setting wanted keys 2024-03-25T19:42:38.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.382+0000 7f57aae7b640 1 librados: calling monclient init 2024-03-25T19:42:38.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.382+0000 7f57aae7b640 10 monclient: init 2024-03-25T19:42:38.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.382+0000 7f57aae7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:38.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.382+0000 7f57aae7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:38.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.382+0000 7f57aae7b640 10 monclient(hunting): picked mon.noname-b con 0x7f57a40b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:38.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.382+0000 7f57aae7b640 10 monclient(hunting): picked mon.noname-a con 0x7f57a40b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:38.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.382+0000 7f57aae7b640 10 monclient(hunting): picked mon.noname-c con 0x7f57a414be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:38.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.382+0000 7f57aae7b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a4136250 con 0x7f57a40b4d60 2024-03-25T19:42:38.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.383+0000 7f57aae7b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a4136af0 con 0x7f57a40b1730 2024-03-25T19:42:38.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.383+0000 7f57aae7b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a4136c80 con 0x7f57a414be90 2024-03-25T19:42:38.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.383+0000 7f57aae7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:38.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.383+0000 7f57aae7b640 10 monclient(hunting): authenticate will time out at 4950.723791s 2024-03-25T19:42:38.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.395+0000 7f57a9418640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f57a414be90 0x7f57a41e2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55990/0 (socket says 172.21.15.107:55990) 2024-03-25T19:42:38.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.395+0000 7f57a9418640 1 -- 172.21.15.107:0/148470487 learned_addr learned my addr 172.21.15.107:0/148470487 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:38.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.397+0000 7f57a17fa640 10 client.?.objecter ms_handle_connect 0x7f57a414be90 2024-03-25T19:42:38.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.397+0000 7f57a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:38.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.402+0000 7f57a17fa640 10 client.?.objecter ms_handle_connect 0x7f57a40b4d60 2024-03-25T19:42:38.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.402+0000 7f57a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:38.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.406+0000 7f57a17fa640 10 client.?.objecter ms_handle_connect 0x7f57a40b1730 2024-03-25T19:42:38.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.406+0000 7f57a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:38.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.409+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812284074 0 0) 0x7f57a4136250 con 0x7f57a40b4d60 2024-03-25T19:42:38.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.409+0000 7f57a17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:38.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.409+0000 7f57a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:38.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.409+0000 7f57a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:38.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.409+0000 7f57a17fa640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:42:38.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.409+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5788004eb0 con 0x7f57a40b4d60 2024-03-25T19:42:38.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.415+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 837565213 0 0) 0x7f5788004eb0 con 0x7f57a40b4d60 2024-03-25T19:42:38.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.415+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57a4136250 con 0x7f57a40b4d60 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.415+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5790001c00 con 0x7f57a40b4d60 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.415+0000 7f57a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.415+0000 7f57a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.415+0000 7f57a17fa640 10 monclient(hunting): dump: 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:38.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.415+0000 7f57a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:38.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 905868582 0 0) 0x7f57a4136250 con 0x7f57a40b4d60 2024-03-25T19:42:38.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 >> v1:172.21.15.107:6790/0 conn(0x7f57a414be90 legacy=0x7f57a41e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:38.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 >> v1:172.21.15.195:6789/0 conn(0x7f57a40b1730 legacy=0x7f57a41d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:38.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57a17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:38.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57a17fa640 1 monclient: found mon.a 2024-03-25T19:42:38.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:38.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57a17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:38.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a4138210 con 0x7f57a40b4d60 2024-03-25T19:42:38.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57a17fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:38.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57a17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:38.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:38.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57aae7b640 5 monclient: authenticate success, global_id 4527 2024-03-25T19:42:38.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57aae7b640 10 monclient: _renew_subs 2024-03-25T19:42:38.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57aae7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:38.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57aae7b640 1 -- 172.21.15.107:0/148470487 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a41384c0 con 0x7f57a40b4d60 2024-03-25T19:42:38.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57aae7b640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:38.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57aae7b640 10 monclient: _renew_subs 2024-03-25T19:42:38.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57aae7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:38.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57aae7b640 1 -- 172.21.15.107:0/148470487 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a4145700 con 0x7f57a40b4d60 2024-03-25T19:42:38.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.424+0000 7f57aae7b640 1 librados: init done 2024-03-25T19:42:38.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.425+0000 7f57aae7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:38.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.425+0000 7f57aae7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:38.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.425+0000 7f57aae7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:38.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.425+0000 7f57aae7b640 1 -- 172.21.15.107:0/148470487 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a4135f70 con 0x7f57a40b4d60 2024-03-25T19:42:38.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.746+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57900038a0 con 0x7f57a40b4d60 2024-03-25T19:42:38.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.746+0000 7f57a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:38.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.746+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5790001250 con 0x7f57a40b4d60 2024-03-25T19:42:38.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.746+0000 7f57a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:38.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.746+0000 7f57a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:38.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.746+0000 7f57a17fa640 10 monclient: dump: 2024-03-25T19:42:38.746 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:38.746 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:38.746 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:38.746 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:38.747 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:38.747 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:38.747 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:38.747 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:38.747 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:38.747 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:38.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.746+0000 7f57a2ffd640 4 set_mon_vals no callback set 2024-03-25T19:42:38.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.747+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f5790003bd0 con 0x7f57a40b4d60 2024-03-25T19:42:38.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.747+0000 7f57a8c17640 10 monclient: get_auth_request con 0x7f578806dc70 auth_method 0 2024-03-25T19:42:38.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.748+0000 7f57a17fa640 10 client.4527.objecter ms_handle_connect 0x7f578806dc70 2024-03-25T19:42:38.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.754+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f5790053570 con 0x7f57a40b4d60 2024-03-25T19:42:38.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.754+0000 7f57a17fa640 10 client.4527.objecter ms_dispatch 0x7f57a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:38.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.754+0000 7f57a17fa640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:38.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.754+0000 7f57a17fa640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:38.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.754+0000 7f57a17fa640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-25T19:42:38.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.754+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f579004eeb0 con 0x7f57a40b4d60 2024-03-25T19:42:38.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.754+0000 7f57a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:38.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:38.754+0000 7f57a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:39.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.007+0000 7f57aae7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:39.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.007+0000 7f57aae7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:39.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.007+0000 7f57aae7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:42:39.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.007+0000 7f57aae7b640 1 -- 172.21.15.107:0/148470487 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57a40627b0 con 0x7f57a40b4d60 2024-03-25T19:42:39.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.065+0000 7f57a17fa640 1 -- 172.21.15.107:0/148470487 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f5790050fb0 con 0x7f57a40b4d60 2024-03-25T19:42:39.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.065+0000 7f57a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:39.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.065+0000 7f57a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:39.065 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:42:39.066 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.067+0000 7f57aae7b640 10 librados: watch_flush enter 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.067+0000 7f57aae7b640 10 librados: watch_flush exit 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.067+0000 7f57aae7b640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.067+0000 7f57aae7b640 10 client.4527.objecter successfully canceled tick 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.068+0000 7f57aae7b640 1 -- 172.21.15.107:0/148470487 >> v1:172.21.15.195:6800/2984408421 conn(0x7f578806dc70 legacy=0x7f5788070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.068+0000 7f57aae7b640 10 monclient: shutdown 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.068+0000 7f57aae7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.068+0000 7f57aae7b640 1 -- 172.21.15.107:0/148470487 >> v1:172.21.15.107:6789/0 conn(0x7f57a40b4d60 legacy=0x7f57a412fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.068+0000 7f57aae7b640 1 -- 172.21.15.107:0/148470487 shutdown_connections 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.068+0000 7f57aae7b640 1 -- 172.21.15.107:0/148470487 >> 172.21.15.107:0/148470487 conn(0x7f57a40617a0 msgr2=0x7f57a40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.068+0000 7f57aae7b640 1 -- 172.21.15.107:0/148470487 shutdown_connections 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.068+0000 7f57aae7b640 1 -- 172.21.15.107:0/148470487 wait complete. 2024-03-25T19:42:39.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:39.068+0000 7f57aae7b640 1 librados: shutdown 2024-03-25T19:42:39.079 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:45.483 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:07:15.799 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:45.499 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:07:15.815 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:45.533 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:07:15.849 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:45.544 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:07:15.860 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:45.548 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:07:15.864 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:45.557 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:07:15.844 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:45.560 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:07:15.847 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:45.583 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:07:15.899 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:45.584 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:07:15.871 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:45.885 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:46.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.002+0000 7f2a92f82640 10 monclient: get_monmap_and_config 2024-03-25T19:42:46.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.002+0000 7f2a92f82640 10 monclient: build_initial_monmap 2024-03-25T19:42:46.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.002+0000 7f2a92f82640 10 monclient: monmap: 2024-03-25T19:42:46.002 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:46.002 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:46.002 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:42:46.003124+0000 2024-03-25T19:42:46.002 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:42:46.003124+0000 2024-03-25T19:42:46.002 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:46.002 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:46.002 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:46.005 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:42:46.005 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:42:46.005 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:46.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.003+0000 7f2a92f82640 1 Processor -- start 2024-03-25T19:42:46.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.003+0000 7f2a92f82640 1 -- start start 2024-03-25T19:42:46.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.003+0000 7f2a92f82640 10 monclient: init 2024-03-25T19:42:46.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.003+0000 7f2a92f82640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:46.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.003+0000 7f2a92f82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:46.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.003+0000 7f2a92f82640 10 monclient(hunting): picked mon.noname-c con 0x7f2a8c0a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:46.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.003+0000 7f2a92f82640 10 monclient(hunting): picked mon.noname-b con 0x7f2a8c13f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:46.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.004+0000 7f2a92f82640 10 monclient(hunting): picked mon.noname-a con 0x7f2a8c1430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:46.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.004+0000 7f2a92f82640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c142520 con 0x7f2a8c1430c0 2024-03-25T19:42:46.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.004+0000 7f2a92f82640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c0b28a0 con 0x7f2a8c0a98f0 2024-03-25T19:42:46.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.004+0000 7f2a92f82640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c0b2a30 con 0x7f2a8c13f710 2024-03-25T19:42:46.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.004+0000 7f2a92f82640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:46.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.004+0000 7f2a92f82640 10 monclient(hunting): authenticate will time out at 4958.344809s 2024-03-25T19:42:46.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.011+0000 7f2a90d1e640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2a8c0a98f0 0x7f2a8c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:48004/0 (socket says 172.21.15.107:48004) 2024-03-25T19:42:46.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.011+0000 7f2a90d1e640 1 -- 172.21.15.107:0/3163686823 learned_addr learned my addr 172.21.15.107:0/3163686823 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:46.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.023+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196950003 0 0) 0x7f2a8c142520 con 0x7f2a8c1430c0 2024-03-25T19:42:46.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.023+0000 7f2a8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:46.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.023+0000 7f2a8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:46.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.023+0000 7f2a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:46.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.023+0000 7f2a8b7fe640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:42:46.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.023+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a7c003520 con 0x7f2a8c1430c0 2024-03-25T19:42:46.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979656920 0 0) 0x7f2a8c0b28a0 con 0x7f2a8c0a98f0 2024-03-25T19:42:46.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:46.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:46.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:46.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:42:46.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a8c142520 con 0x7f2a8c0a98f0 2024-03-25T19:42:46.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881514443 0 0) 0x7f2a8c0b2a30 con 0x7f2a8c13f710 2024-03-25T19:42:46.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:46.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:46.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:46.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:42:46.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.027+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a8c0b28a0 con 0x7f2a8c13f710 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 581092890 0 0) 0x7f2a8c0b28a0 con 0x7f2a8c13f710 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 >> v1:172.21.15.195:6789/0 conn(0x7f2a8c0a98f0 legacy=0x7f2a8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 >> v1:172.21.15.107:6789/0 conn(0x7f2a8c1430c0 legacy=0x7f2a8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 1 monclient: found mon.noname-b 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c0b44c0 con 0x7f2a8c13f710 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:46.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2a780015b0 con 0x7f2a8c13f710 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a92f82640 5 monclient: authenticate success, global_id 4535 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a92f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a8b7fe640 10 monclient: dump: 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:46.040 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:46.041 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:46.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.039+0000 7f2a92f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:46.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.045+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a78001710 con 0x7f2a8c13f710 2024-03-25T19:42:46.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.045+0000 7f2a8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.045+0000 7f2a8b7fe640 1 -- 172.21.15.107:0/3163686823 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2a78001b60 con 0x7f2a8c13f710 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.045+0000 7f2a8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.045+0000 7f2a8b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.045+0000 7f2a8b7fe640 10 monclient: dump: 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:46.045 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:46.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.045+0000 7f2a92f82640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:46.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.045+0000 7f2a92f82640 4 set_mon_vals no callback set 2024-03-25T19:42:46.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.046+0000 7f2a92f82640 10 monclient: shutdown 2024-03-25T19:42:46.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.046+0000 7f2a92f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:46.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.046+0000 7f2a92f82640 1 -- 172.21.15.107:0/3163686823 >> v1:172.21.15.107:6790/0 conn(0x7f2a8c13f710 legacy=0x7f2a8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.046+0000 7f2a92f82640 1 -- 172.21.15.107:0/3163686823 shutdown_connections 2024-03-25T19:42:46.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.046+0000 7f2a92f82640 1 -- 172.21.15.107:0/3163686823 >> 172.21.15.107:0/3163686823 conn(0x7f2a8c0b7ea0 msgr2=0x7f2a8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:46.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.046+0000 7f2a92f82640 1 -- 172.21.15.107:0/3163686823 shutdown_connections 2024-03-25T19:42:46.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.046+0000 7f2a92f82640 1 -- 172.21.15.107:0/3163686823 wait complete. 2024-03-25T19:42:46.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.047+0000 7f2a92f82640 10 monclient: build_initial_monmap 2024-03-25T19:42:46.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.047+0000 7f2a92f82640 10 monclient: monmap: 2024-03-25T19:42:46.047 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:46.047 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.047+0000 7f2a92f82640 1 librados: starting msgr at 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.047+0000 7f2a92f82640 1 librados: starting objecter 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.047+0000 7f2a92f82640 1 Processor -- start 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.047+0000 7f2a92f82640 1 -- start start 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.047+0000 7f2a92f82640 1 librados: setting wanted keys 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.047+0000 7f2a92f82640 1 librados: calling monclient init 2024-03-25T19:42:46.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.047+0000 7f2a92f82640 10 monclient: init 2024-03-25T19:42:46.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.048+0000 7f2a92f82640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:46.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.048+0000 7f2a92f82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:46.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.048+0000 7f2a92f82640 10 monclient(hunting): picked mon.noname-c con 0x7f2a8c0a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:46.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.048+0000 7f2a92f82640 10 monclient(hunting): picked mon.noname-a con 0x7f2a8c13f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:46.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.048+0000 7f2a92f82640 10 monclient(hunting): picked mon.noname-b con 0x7f2a8c1430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:46.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.048+0000 7f2a92f82640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c139d20 con 0x7f2a8c13f710 2024-03-25T19:42:46.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.048+0000 7f2a92f82640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c139fc0 con 0x7f2a8c1430c0 2024-03-25T19:42:46.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.048+0000 7f2a92f82640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c0b79b0 con 0x7f2a8c0a98f0 2024-03-25T19:42:46.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.048+0000 7f2a92f82640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:46.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.048+0000 7f2a92f82640 10 monclient(hunting): authenticate will time out at 4958.389570s 2024-03-25T19:42:46.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.058+0000 7f2a9151f640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2a8c1430c0 0x7f2a8c1e6750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:48026/0 (socket says 172.21.15.107:48026) 2024-03-25T19:42:46.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.058+0000 7f2a9151f640 1 -- 172.21.15.107:0/4188194608 learned_addr learned my addr 172.21.15.107:0/4188194608 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:46.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.060+0000 7f2a897fa640 10 client.?.objecter ms_handle_connect 0x7f2a8c1430c0 2024-03-25T19:42:46.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.060+0000 7f2a897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:46.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.063+0000 7f2a897fa640 10 client.?.objecter ms_handle_connect 0x7f2a8c0a98f0 2024-03-25T19:42:46.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.063+0000 7f2a897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:46.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.067+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581123861 0 0) 0x7f2a8c139fc0 con 0x7f2a8c1430c0 2024-03-25T19:42:46.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.067+0000 7f2a897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:46.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.067+0000 7f2a897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:46.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.067+0000 7f2a897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:46.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.067+0000 7f2a897fa640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:42:46.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.067+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a70004eb0 con 0x7f2a8c1430c0 2024-03-25T19:42:46.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.070+0000 7f2a897fa640 10 client.?.objecter ms_handle_connect 0x7f2a8c13f710 2024-03-25T19:42:46.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.070+0000 7f2a897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:46.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.078+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1718571789 0 0) 0x7f2a70004eb0 con 0x7f2a8c1430c0 2024-03-25T19:42:46.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.078+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a8c139fc0 con 0x7f2a8c1430c0 2024-03-25T19:42:46.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.078+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2a84001c30 con 0x7f2a8c1430c0 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.078+0000 7f2a897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.078+0000 7f2a897fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.078+0000 7f2a897fa640 10 monclient(hunting): dump: 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:46.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.078+0000 7f2a897fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:46.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.080+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728668396 0 0) 0x7f2a8c139d20 con 0x7f2a8c13f710 2024-03-25T19:42:46.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.080+0000 7f2a897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:46.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.080+0000 7f2a897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:46.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.080+0000 7f2a897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:46.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.080+0000 7f2a897fa640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:42:46.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.080+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a70004eb0 con 0x7f2a8c13f710 2024-03-25T19:42:46.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.086+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153688840 0 0) 0x7f2a8c0b79b0 con 0x7f2a8c0a98f0 2024-03-25T19:42:46.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.086+0000 7f2a897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:46.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.086+0000 7f2a897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:46.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.086+0000 7f2a897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:46.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.086+0000 7f2a897fa640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:42:46.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.086+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a8c139d20 con 0x7f2a8c0a98f0 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2351369430 0 0) 0x7f2a8c139fc0 con 0x7f2a8c1430c0 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 >> v1:172.21.15.107:6790/0 conn(0x7f2a8c0a98f0 legacy=0x7f2a8c1df1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 >> v1:172.21.15.107:6789/0 conn(0x7f2a8c13f710 legacy=0x7f2a8c1df8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a897fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a897fa640 1 monclient: found mon.b 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a897fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c0b7b40 con 0x7f2a8c1430c0 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a897fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a897fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a92f82640 5 monclient: authenticate success, global_id 4525 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a92f82640 10 monclient: _renew_subs 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a92f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:46.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.087+0000 7f2a92f82640 1 -- 172.21.15.107:0/4188194608 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8c0b7460 con 0x7f2a8c1430c0 2024-03-25T19:42:46.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.088+0000 7f2a92f82640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:46.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.088+0000 7f2a92f82640 10 monclient: _renew_subs 2024-03-25T19:42:46.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.088+0000 7f2a92f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:46.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.088+0000 7f2a92f82640 1 -- 172.21.15.107:0/4188194608 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8c14d540 con 0x7f2a8c1430c0 2024-03-25T19:42:46.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.088+0000 7f2a92f82640 1 librados: init done 2024-03-25T19:42:46.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.088+0000 7f2a92f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:46.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.088+0000 7f2a92f82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:46.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.088+0000 7f2a92f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:46.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.088+0000 7f2a92f82640 1 -- 172.21.15.107:0/4188194608 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8c13a150 con 0x7f2a8c1430c0 2024-03-25T19:42:46.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.370+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a84003970 con 0x7f2a8c1430c0 2024-03-25T19:42:46.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.370+0000 7f2a897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.370+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2a84003dc0 con 0x7f2a8c1430c0 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.370+0000 7f2a897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.370+0000 7f2a897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.370+0000 7f2a897fa640 10 monclient: dump: 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:46.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.370+0000 7f2a8affd640 4 set_mon_vals no callback set 2024-03-25T19:42:46.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.371+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f2a84003fe0 con 0x7f2a8c1430c0 2024-03-25T19:42:46.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.372+0000 7f2a90d1e640 10 monclient: get_auth_request con 0x7f2a7006de80 auth_method 0 2024-03-25T19:42:46.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.372+0000 7f2a897fa640 10 client.4525.objecter ms_handle_connect 0x7f2a7006de80 2024-03-25T19:42:46.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.376+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f2a84054a80 con 0x7f2a8c1430c0 2024-03-25T19:42:46.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.376+0000 7f2a897fa640 10 client.4525.objecter ms_dispatch 0x7f2a8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:46.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.376+0000 7f2a897fa640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:46.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.376+0000 7f2a897fa640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:46.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.376+0000 7f2a897fa640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-25T19:42:46.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.377+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a84050590 con 0x7f2a8c1430c0 2024-03-25T19:42:46.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.377+0000 7f2a897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:46.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.377+0000 7f2a897fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:46.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.588+0000 7f2a92f82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:46.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.588+0000 7f2a92f82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:46.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.588+0000 7f2a92f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:42:46.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.588+0000 7f2a92f82640 1 -- 172.21.15.107:0/4188194608 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a8c0627b0 con 0x7f2a8c1430c0 2024-03-25T19:42:46.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.715+0000 7f2a897fa640 1 -- 172.21.15.107:0/4188194608 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f2a840524c0 con 0x7f2a8c1430c0 2024-03-25T19:42:46.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.715+0000 7f2a897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:46.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.715+0000 7f2a897fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:46.715 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:42:46.715 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:46.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 10 librados: watch_flush enter 2024-03-25T19:42:46.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 10 librados: watch_flush exit 2024-03-25T19:42:46.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-25T19:42:46.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 10 client.4525.objecter successfully canceled tick 2024-03-25T19:42:46.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 1 -- 172.21.15.107:0/4188194608 >> v1:172.21.15.195:6800/2984408421 conn(0x7f2a7006de80 legacy=0x7f2a70070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 10 monclient: shutdown 2024-03-25T19:42:46.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:46.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 1 -- 172.21.15.107:0/4188194608 >> v1:172.21.15.195:6789/0 conn(0x7f2a8c1430c0 legacy=0x7f2a8c1e6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:46.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 1 -- 172.21.15.107:0/4188194608 shutdown_connections 2024-03-25T19:42:46.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 1 -- 172.21.15.107:0/4188194608 >> 172.21.15.107:0/4188194608 conn(0x7f2a8c0b7ea0 msgr2=0x7f2a8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:46.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 1 -- 172.21.15.107:0/4188194608 shutdown_connections 2024-03-25T19:42:46.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 1 -- 172.21.15.107:0/4188194608 wait complete. 2024-03-25T19:42:46.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:46.718+0000 7f2a92f82640 1 librados: shutdown 2024-03-25T19:42:46.729 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:53.536 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.654+0000 7f2df034d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.654+0000 7f2df034d640 10 monclient: build_initial_monmap 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.654+0000 7f2df034d640 10 monclient: monmap: 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:42:53.655184+0000 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:42:53.655184+0000 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:42:53.654 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:53.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.655+0000 7f2df034d640 1 Processor -- start 2024-03-25T19:42:53.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.655+0000 7f2df034d640 1 -- start start 2024-03-25T19:42:53.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.655+0000 7f2df034d640 10 monclient: init 2024-03-25T19:42:53.656 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.656+0000 7f2df034d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:53.656 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.656+0000 7f2df034d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:53.656 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.656+0000 7f2df034d640 10 monclient(hunting): picked mon.noname-a con 0x7f2de80a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:53.656 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.656+0000 7f2df034d640 10 monclient(hunting): picked mon.noname-c con 0x7f2de813f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:53.656 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.656+0000 7f2df034d640 10 monclient(hunting): picked mon.noname-b con 0x7f2de81430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:53.656 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.656+0000 7f2df034d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de8142520 con 0x7f2de80a98f0 2024-03-25T19:42:53.656 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.656+0000 7f2df034d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de80b28a0 con 0x7f2de813f710 2024-03-25T19:42:53.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.656+0000 7f2df034d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de80b2a30 con 0x7f2de81430c0 2024-03-25T19:42:53.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.656+0000 7f2df034d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:53.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.656+0000 7f2df034d640 10 monclient(hunting): authenticate will time out at 4965.997399s 2024-03-25T19:42:53.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.659+0000 7f2ded8e8640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2de813f710 0x7f2de81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:37560/0 (socket says 172.21.15.107:37560) 2024-03-25T19:42:53.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.659+0000 7f2ded8e8640 1 -- 172.21.15.107:0/1618180065 learned_addr learned my addr 172.21.15.107:0/1618180065 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:53.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.672+0000 7f2ded0e7640 1 -- 172.21.15.107:0/1618180065 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194851070 0 0) 0x7f2de80b2a30 con 0x7f2de81430c0 2024-03-25T19:42:53.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.672+0000 7f2ded0e7640 10 monclient(hunting): _negotiate 2024-03-25T19:42:53.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.672+0000 7f2ded0e7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:53.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.672+0000 7f2ded0e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:53.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.672+0000 7f2ded0e7640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:42:53.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.672+0000 7f2ded0e7640 1 -- 172.21.15.107:0/1618180065 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd4003520 con 0x7f2de81430c0 2024-03-25T19:42:53.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.673+0000 7f2ded0e7640 1 -- 172.21.15.107:0/1618180065 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967780072 0 0) 0x7f2de80b28a0 con 0x7f2de813f710 2024-03-25T19:42:53.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.673+0000 7f2ded0e7640 10 monclient(hunting): _negotiate 2024-03-25T19:42:53.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.673+0000 7f2ded0e7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:53.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.673+0000 7f2ded0e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:53.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.673+0000 7f2ded0e7640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:42:53.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.673+0000 7f2ded0e7640 1 -- 172.21.15.107:0/1618180065 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de80b2a30 con 0x7f2de813f710 2024-03-25T19:42:53.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 1 -- 172.21.15.107:0/1618180065 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1791592127 0 0) 0x7f2dd4003520 con 0x7f2de81430c0 2024-03-25T19:42:53.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 1 -- 172.21.15.107:0/1618180065 >> v1:172.21.15.195:6789/0 conn(0x7f2de813f710 legacy=0x7f2de81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 1 -- 172.21.15.107:0/1618180065 >> v1:172.21.15.107:6789/0 conn(0x7f2de80a98f0 legacy=0x7f2de80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 10 monclient: _finish_hunting 0 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 1 monclient: found mon.noname-b 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 1 -- 172.21.15.107:0/1618180065 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de80b44c0 con 0x7f2de81430c0 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 10 monclient: _finish_auth 0 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 10 monclient: _check_auth_tickets 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 1 -- 172.21.15.107:0/1618180065 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2de0001a10 con 0x7f2de81430c0 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2ded0e7640 10 monclient: dump: 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:53.683 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:53.684 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:53.684 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:53.684 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:53.684 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:53.684 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:53.684 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:53.684 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2df034d640 5 monclient: authenticate success, global_id 4541 2024-03-25T19:42:53.684 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.682+0000 7f2df034d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:53.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.703+0000 7f2ded0e7640 1 -- 172.21.15.107:0/1618180065 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2de0001b70 con 0x7f2de81430c0 2024-03-25T19:42:53.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.703+0000 7f2ded0e7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:53.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.703+0000 7f2ded0e7640 1 -- 172.21.15.107:0/1618180065 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2de0001f80 con 0x7f2de81430c0 2024-03-25T19:42:53.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.703+0000 7f2ded0e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:53.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.703+0000 7f2ded0e7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:53.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.703+0000 7f2ded0e7640 10 monclient: dump: 2024-03-25T19:42:53.703 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:53.703 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:53.703 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:53.703 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:53.704 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:53.704 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:53.704 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:53.704 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:53.704 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:53.704 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:53.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.703+0000 7f2df034d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:53.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.703+0000 7f2df034d640 4 set_mon_vals no callback set 2024-03-25T19:42:53.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.704+0000 7f2df034d640 10 monclient: shutdown 2024-03-25T19:42:53.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.704+0000 7f2df034d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:53.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.704+0000 7f2df034d640 1 -- 172.21.15.107:0/1618180065 >> v1:172.21.15.107:6790/0 conn(0x7f2de81430c0 legacy=0x7f2de814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.704+0000 7f2df034d640 1 -- 172.21.15.107:0/1618180065 shutdown_connections 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.704+0000 7f2df034d640 1 -- 172.21.15.107:0/1618180065 >> 172.21.15.107:0/1618180065 conn(0x7f2de80b7ea0 msgr2=0x7f2de80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.704+0000 7f2df034d640 1 -- 172.21.15.107:0/1618180065 shutdown_connections 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.704+0000 7f2df034d640 1 -- 172.21.15.107:0/1618180065 wait complete. 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.705+0000 7f2df034d640 10 monclient: build_initial_monmap 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.705+0000 7f2df034d640 10 monclient: monmap: 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:42:53.706 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:42:53.707 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:42:53.707 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:53.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.705+0000 7f2df034d640 1 librados: starting msgr at 2024-03-25T19:42:53.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.705+0000 7f2df034d640 1 librados: starting objecter 2024-03-25T19:42:53.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.705+0000 7f2df034d640 1 Processor -- start 2024-03-25T19:42:53.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.705+0000 7f2df034d640 1 -- start start 2024-03-25T19:42:53.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.705+0000 7f2df034d640 1 librados: setting wanted keys 2024-03-25T19:42:53.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.705+0000 7f2df034d640 1 librados: calling monclient init 2024-03-25T19:42:53.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.705+0000 7f2df034d640 10 monclient: init 2024-03-25T19:42:53.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.706+0000 7f2df034d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:53.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.706+0000 7f2df034d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:53.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.706+0000 7f2df034d640 10 monclient(hunting): picked mon.noname-c con 0x7f2de80a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:42:53.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.706+0000 7f2df034d640 10 monclient(hunting): picked mon.noname-a con 0x7f2de813f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:42:53.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.706+0000 7f2df034d640 10 monclient(hunting): picked mon.noname-b con 0x7f2de81430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:42:53.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.706+0000 7f2df034d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de8136ad0 con 0x7f2de813f710 2024-03-25T19:42:53.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.706+0000 7f2df034d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de80b7520 con 0x7f2de81430c0 2024-03-25T19:42:53.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.706+0000 7f2df034d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de8135810 con 0x7f2de80a98f0 2024-03-25T19:42:53.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.706+0000 7f2df034d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:53.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.706+0000 7f2df034d640 10 monclient(hunting): authenticate will time out at 4966.047555s 2024-03-25T19:42:53.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.713+0000 7f2dee0e9640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f2de80a98f0 0x7f2de8135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54920/0 (socket says 172.21.15.107:54920) 2024-03-25T19:42:53.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.713+0000 7f2dee0e9640 1 -- 172.21.15.107:0/1777797763 learned_addr learned my addr 172.21.15.107:0/1777797763 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:42:53.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.719+0000 7f2ddeffd640 10 client.?.objecter ms_handle_connect 0x7f2de81430c0 2024-03-25T19:42:53.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.719+0000 7f2ddeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:53.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.720+0000 7f2ddeffd640 10 client.?.objecter ms_handle_connect 0x7f2de80a98f0 2024-03-25T19:42:53.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.720+0000 7f2ddeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:53.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761712611 0 0) 0x7f2de80b7520 con 0x7f2de81430c0 2024-03-25T19:42:53.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:53.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:53.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:53.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:42:53.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dcc004eb0 con 0x7f2de81430c0 2024-03-25T19:42:53.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 10 client.?.objecter ms_handle_connect 0x7f2de813f710 2024-03-25T19:42:53.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:53.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127640978 0 0) 0x7f2de8135810 con 0x7f2de80a98f0 2024-03-25T19:42:53.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:53.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:53.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:53.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:42:53.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.728+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de80b7520 con 0x7f2de80a98f0 2024-03-25T19:42:53.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.743+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1494759517 0 0) 0x7f2de80b7520 con 0x7f2de80a98f0 2024-03-25T19:42:53.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.743+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2de8135810 con 0x7f2de80a98f0 2024-03-25T19:42:53.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.743+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2de40019b0 con 0x7f2de80a98f0 2024-03-25T19:42:53.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.743+0000 7f2ddeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:53.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.743+0000 7f2ddeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:53.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.743+0000 7f2ddeffd640 10 monclient(hunting): dump: 2024-03-25T19:42:53.743 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:53.744 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:53.744 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:53.744 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:53.744 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:53.744 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:53.744 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:53.744 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:53.744 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:53.744 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:53.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.743+0000 7f2ddeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 262290770 0 0) 0x7f2de8135810 con 0x7f2de80a98f0 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 >> v1:172.21.15.195:6789/0 conn(0x7f2de81430c0 legacy=0x7f2de8130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 >> v1:172.21.15.107:6789/0 conn(0x7f2de813f710 legacy=0x7f2de80b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2ddeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2ddeffd640 1 monclient: found mon.c 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2ddeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2ddeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de80b6180 con 0x7f2de80a98f0 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2ddeffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2ddeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2ddeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2df034d640 5 monclient: authenticate success, global_id 4544 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2df034d640 10 monclient: _renew_subs 2024-03-25T19:42:53.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2df034d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:42:53.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2df034d640 1 -- 172.21.15.107:0/1777797763 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de80b7a40 con 0x7f2de80a98f0 2024-03-25T19:42:53.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2df034d640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:53.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2df034d640 10 monclient: _renew_subs 2024-03-25T19:42:53.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2df034d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:42:53.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2df034d640 1 -- 172.21.15.107:0/1777797763 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de8131840 con 0x7f2de80a98f0 2024-03-25T19:42:53.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.751+0000 7f2df034d640 1 librados: init done 2024-03-25T19:42:53.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.752+0000 7f2df034d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:53.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.752+0000 7f2df034d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:53.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.752+0000 7f2df034d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:42:53.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:53.752+0000 7f2df034d640 1 -- 172.21.15.107:0/1777797763 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de8136f60 con 0x7f2de80a98f0 2024-03-25T19:42:54.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.082+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2de4003610 con 0x7f2de80a98f0 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.082+0000 7f2ddeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.082+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2de4003780 con 0x7f2de80a98f0 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.082+0000 7f2ddeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.082+0000 7f2ddeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.082+0000 7f2ddeffd640 10 monclient: dump: 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:42:54.082 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:42:54.083 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:42:54.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.082+0000 7f2dec8e6640 4 set_mon_vals no callback set 2024-03-25T19:42:54.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.082+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f2de40039a0 con 0x7f2de80a98f0 2024-03-25T19:42:54.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.083+0000 7f2ded8e8640 10 monclient: get_auth_request con 0x7f2dcc06dd90 auth_method 0 2024-03-25T19:42:54.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.083+0000 7f2ddeffd640 10 client.4544.objecter ms_handle_connect 0x7f2dcc06dd90 2024-03-25T19:42:54.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.089+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f2de4054600 con 0x7f2de80a98f0 2024-03-25T19:42:54.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.089+0000 7f2ddeffd640 10 client.4544.objecter ms_dispatch 0x7f2de8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:54.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.089+0000 7f2ddeffd640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:54.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.089+0000 7f2ddeffd640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:54.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.089+0000 7f2ddeffd640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-25T19:42:54.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.089+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2de4050110 con 0x7f2de80a98f0 2024-03-25T19:42:54.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.089+0000 7f2ddeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:54.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.089+0000 7f2ddeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:54.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.342+0000 7f2df034d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:54.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.342+0000 7f2df034d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:54.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.342+0000 7f2df034d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:42:54.342 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.342+0000 7f2df034d640 1 -- 172.21.15.107:0/1777797763 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2de80b76b0 con 0x7f2de80a98f0 2024-03-25T19:42:54.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.416+0000 7f2ddeffd640 1 -- 172.21.15.107:0/1777797763 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f2de4052040 con 0x7f2de80a98f0 2024-03-25T19:42:54.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.416+0000 7f2ddeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:54.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.416+0000 7f2ddeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:54.416 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:42:54.416 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:54.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 10 librados: watch_flush enter 2024-03-25T19:42:54.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 10 librados: watch_flush exit 2024-03-25T19:42:54.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 10 client.4544.objecter successfully canceled tick 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 1 -- 172.21.15.107:0/1777797763 >> v1:172.21.15.195:6800/2984408421 conn(0x7f2dcc06dd90 legacy=0x7f2dcc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 10 monclient: shutdown 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 1 -- 172.21.15.107:0/1777797763 >> v1:172.21.15.107:6790/0 conn(0x7f2de80a98f0 legacy=0x7f2de8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 1 -- 172.21.15.107:0/1777797763 shutdown_connections 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 1 -- 172.21.15.107:0/1777797763 >> 172.21.15.107:0/1777797763 conn(0x7f2de80b7ea0 msgr2=0x7f2de8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 1 -- 172.21.15.107:0/1777797763 shutdown_connections 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.418+0000 7f2df034d640 1 -- 172.21.15.107:0/1777797763 wait complete. 2024-03-25T19:42:54.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:42:54.419+0000 7f2df034d640 1 librados: shutdown 2024-03-25T19:42:54.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:01.236 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:01.345 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.346+0000 7f22930c8640 10 monclient: get_monmap_and_config 2024-03-25T19:43:01.345 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.346+0000 7f22930c8640 10 monclient: build_initial_monmap 2024-03-25T19:43:01.345 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.346+0000 7f22930c8640 10 monclient: monmap: 2024-03-25T19:43:01.346 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:01.346 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:01.346 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:43:01.347027+0000 2024-03-25T19:43:01.346 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:43:01.347027+0000 2024-03-25T19:43:01.346 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:01.346 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:01.346 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:01.346 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:43:01.346 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:43:01.346 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:01.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.346+0000 7f22930c8640 1 Processor -- start 2024-03-25T19:43:01.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 1 -- start start 2024-03-25T19:43:01.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 10 monclient: init 2024-03-25T19:43:01.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:01.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:01.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 10 monclient(hunting): picked mon.noname-b con 0x7f228c0b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:01.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 10 monclient(hunting): picked mon.noname-c con 0x7f228c0b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:01.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 10 monclient(hunting): picked mon.noname-a con 0x7f228c14be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:01.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c0af940 con 0x7f228c14be90 2024-03-25T19:43:01.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c0b0a70 con 0x7f228c0b1730 2024-03-25T19:43:01.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c0b0d50 con 0x7f228c0b4d60 2024-03-25T19:43:01.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:01.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.347+0000 7f22930c8640 10 monclient(hunting): authenticate will time out at 4973.688551s 2024-03-25T19:43:01.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.357+0000 7f2290e64640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f228c0b4d60 0x7f228c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37594/0 (socket says 172.21.15.107:37594) 2024-03-25T19:43:01.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.358+0000 7f2290e64640 1 -- 172.21.15.107:0/4245049066 learned_addr learned my addr 172.21.15.107:0/4245049066 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:01.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.366+0000 7f228b7fe640 1 -- 172.21.15.107:0/4245049066 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364816797 0 0) 0x7f228c0b0d50 con 0x7f228c0b4d60 2024-03-25T19:43:01.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.367+0000 7f228b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:01.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.367+0000 7f228b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:01.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.367+0000 7f228b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:01.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.367+0000 7f228b7fe640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:43:01.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.367+0000 7f228b7fe640 1 -- 172.21.15.107:0/4245049066 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f227c003520 con 0x7f228c0b4d60 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.376+0000 7f228b7fe640 1 -- 172.21.15.107:0/4245049066 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794921278 0 0) 0x7f228c0af940 con 0x7f228c14be90 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.376+0000 7f228b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.376+0000 7f228b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.376+0000 7f228b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:01.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.376+0000 7f228b7fe640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:43:01.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 1 -- 172.21.15.107:0/4245049066 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c0b0d50 con 0x7f228c14be90 2024-03-25T19:43:01.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 1 -- 172.21.15.107:0/4245049066 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 189793320 0 0) 0x7f227c003520 con 0x7f228c0b4d60 2024-03-25T19:43:01.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 1 -- 172.21.15.107:0/4245049066 >> v1:172.21.15.195:6789/0 conn(0x7f228c0b1730 legacy=0x7f228c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:01.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 1 -- 172.21.15.107:0/4245049066 >> v1:172.21.15.107:6789/0 conn(0x7f228c14be90 legacy=0x7f228c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:01.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:01.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 1 monclient: found mon.noname-b 2024-03-25T19:43:01.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 1 -- 172.21.15.107:0/4245049066 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c0b1030 con 0x7f228c0b4d60 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f22930c8640 5 monclient: authenticate success, global_id 4547 2024-03-25T19:43:01.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 1 -- 172.21.15.107:0/4245049066 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2280001690 con 0x7f228c0b4d60 2024-03-25T19:43:01.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f22930c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:01.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:01.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:01.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f228b7fe640 10 monclient: dump: 2024-03-25T19:43:01.381 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:01.381 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:01.381 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.381 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.382 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:01.382 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:01.382 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:01.382 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:01.382 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:01.382 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:01.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.377+0000 7f22930c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:01.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.393+0000 7f228b7fe640 1 -- 172.21.15.107:0/4245049066 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22800017f0 con 0x7f228c0b4d60 2024-03-25T19:43:01.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.393+0000 7f228b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.393+0000 7f228b7fe640 1 -- 172.21.15.107:0/4245049066 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2280001c00 con 0x7f228c0b4d60 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.393+0000 7f228b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.393+0000 7f228b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.393+0000 7f228b7fe640 10 monclient: dump: 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:01.393 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:01.394 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:01.394 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:01.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.393+0000 7f22930c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:01.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.393+0000 7f22930c8640 4 set_mon_vals no callback set 2024-03-25T19:43:01.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.394+0000 7f22930c8640 10 monclient: shutdown 2024-03-25T19:43:01.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.394+0000 7f22930c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:01.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.394+0000 7f22930c8640 1 -- 172.21.15.107:0/4245049066 >> v1:172.21.15.107:6790/0 conn(0x7f228c0b4d60 legacy=0x7f228c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:01.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.394+0000 7f22930c8640 1 -- 172.21.15.107:0/4245049066 shutdown_connections 2024-03-25T19:43:01.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.394+0000 7f22930c8640 1 -- 172.21.15.107:0/4245049066 >> 172.21.15.107:0/4245049066 conn(0x7f228c0617a0 msgr2=0x7f228c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:01.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.394+0000 7f22930c8640 1 -- 172.21.15.107:0/4245049066 shutdown_connections 2024-03-25T19:43:01.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.394+0000 7f22930c8640 1 -- 172.21.15.107:0/4245049066 wait complete. 2024-03-25T19:43:01.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.394+0000 7f22930c8640 10 monclient: build_initial_monmap 2024-03-25T19:43:01.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.394+0000 7f22930c8640 10 monclient: monmap: 2024-03-25T19:43:01.398 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:01.398 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:01.398 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:43:01.398 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:43:01.398 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:01.398 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:43:01.398 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:01.398 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:43:01.399 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:43:01.399 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.395+0000 7f22930c8640 1 librados: starting msgr at 2024-03-25T19:43:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.395+0000 7f22930c8640 1 librados: starting objecter 2024-03-25T19:43:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.395+0000 7f22930c8640 1 Processor -- start 2024-03-25T19:43:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.395+0000 7f22930c8640 1 -- start start 2024-03-25T19:43:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.395+0000 7f22930c8640 1 librados: setting wanted keys 2024-03-25T19:43:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.395+0000 7f22930c8640 1 librados: calling monclient init 2024-03-25T19:43:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.395+0000 7f22930c8640 10 monclient: init 2024-03-25T19:43:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.396+0000 7f22930c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.396+0000 7f22930c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.396+0000 7f22930c8640 10 monclient(hunting): picked mon.noname-a con 0x7f228c0b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.396+0000 7f22930c8640 10 monclient(hunting): picked mon.noname-b con 0x7f228c0b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.396+0000 7f22930c8640 10 monclient(hunting): picked mon.noname-c con 0x7f228c14be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.396+0000 7f22930c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c1376f0 con 0x7f228c0b1730 2024-03-25T19:43:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.396+0000 7f22930c8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c1443c0 con 0x7f228c0b4d60 2024-03-25T19:43:01.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.396+0000 7f22930c8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c144550 con 0x7f228c14be90 2024-03-25T19:43:01.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.396+0000 7f22930c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:01.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.396+0000 7f22930c8640 10 monclient(hunting): authenticate will time out at 4973.737494s 2024-03-25T19:43:01.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.401+0000 7f2290e64640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f228c0b1730 0x7f228c12fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40010/0 (socket says 172.21.15.107:40010) 2024-03-25T19:43:01.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.401+0000 7f2290e64640 1 -- 172.21.15.107:0/4159667947 learned_addr learned my addr 172.21.15.107:0/4159667947 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:01.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.403+0000 7f22897fa640 10 client.?.objecter ms_handle_connect 0x7f228c0b1730 2024-03-25T19:43:01.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.403+0000 7f22897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:01.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.405+0000 7f22897fa640 10 client.?.objecter ms_handle_connect 0x7f228c14be90 2024-03-25T19:43:01.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.405+0000 7f22897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:01.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.407+0000 7f22897fa640 10 client.?.objecter ms_handle_connect 0x7f228c0b4d60 2024-03-25T19:43:01.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.407+0000 7f22897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:01.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.412+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705882198 0 0) 0x7f228c1376f0 con 0x7f228c0b1730 2024-03-25T19:43:01.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.412+0000 7f22897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:01.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.412+0000 7f22897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:01.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.412+0000 7f22897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:01.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.412+0000 7f22897fa640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:43:01.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.412+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2270004eb0 con 0x7f228c0b1730 2024-03-25T19:43:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.422+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100722258 0 0) 0x7f228c144550 con 0x7f228c14be90 2024-03-25T19:43:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.422+0000 7f22897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.422+0000 7f22897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.422+0000 7f22897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.422+0000 7f22897fa640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:43:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.422+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c1376f0 con 0x7f228c14be90 2024-03-25T19:43:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.424+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513508585 0 0) 0x7f228c1443c0 con 0x7f228c0b4d60 2024-03-25T19:43:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.424+0000 7f22897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.424+0000 7f22897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.424+0000 7f22897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.424+0000 7f22897fa640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:43:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.424+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c144550 con 0x7f228c0b4d60 2024-03-25T19:43:01.430 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.431+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3805410461 0 0) 0x7f228c1376f0 con 0x7f228c14be90 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.431+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f228c1443c0 con 0x7f228c14be90 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.431+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2284001b90 con 0x7f228c14be90 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.431+0000 7f22897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.431+0000 7f22897fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.431+0000 7f22897fa640 10 monclient(hunting): dump: 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:01.431 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:01.432 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:01.432 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:01.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.431+0000 7f22897fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:01.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.434+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1349661198 0 0) 0x7f228c144550 con 0x7f228c0b4d60 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.434+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f228c1376f0 con 0x7f228c0b4d60 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.434+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2278001830 con 0x7f228c0b4d60 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.434+0000 7f22897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.434+0000 7f22897fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.434+0000 7f22897fa640 10 monclient(hunting): dump: 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:01.434 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:01.435 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:01.435 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:01.435 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:01.435 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.439+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1987055355 0 0) 0x7f2270004eb0 con 0x7f228c0b1730 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.439+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f228c144550 con 0x7f228c0b1730 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.439+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2280001b40 con 0x7f228c0b1730 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.439+0000 7f22897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.439+0000 7f22897fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.439+0000 7f22897fa640 10 monclient(hunting): dump: 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:01.439 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:01.440 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.440 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.440 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:01.440 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:01.440 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:01.440 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:01.440 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:01.440 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2645300775 0 0) 0x7f228c1443c0 con 0x7f228c14be90 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 >> v1:172.21.15.195:6789/0 conn(0x7f228c0b4d60 legacy=0x7f228c1d76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 >> v1:172.21.15.107:6789/0 conn(0x7f228c0b1730 legacy=0x7f228c12fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22897fa640 10 monclient: _finish_hunting 0 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22897fa640 1 monclient: found mon.c 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22897fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c137210 con 0x7f228c14be90 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22897fa640 10 monclient: _finish_auth 0 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22897fa640 10 monclient: _check_auth_tickets 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22930c8640 5 monclient: authenticate success, global_id 4550 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22930c8640 10 monclient: _renew_subs 2024-03-25T19:43:01.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22930c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:01.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22930c8640 1 -- 172.21.15.107:0/4159667947 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c137460 con 0x7f228c14be90 2024-03-25T19:43:01.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22930c8640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:01.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22930c8640 10 monclient: _renew_subs 2024-03-25T19:43:01.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22930c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:01.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22930c8640 1 -- 172.21.15.107:0/4159667947 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c1440c0 con 0x7f228c14be90 2024-03-25T19:43:01.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.448+0000 7f22930c8640 1 librados: init done 2024-03-25T19:43:01.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.449+0000 7f22930c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:01.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.449+0000 7f22930c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:01.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.449+0000 7f22930c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:01.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.449+0000 7f22930c8640 1 -- 172.21.15.107:0/4159667947 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c137930 con 0x7f228c14be90 2024-03-25T19:43:01.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.758+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2284002040 con 0x7f228c14be90 2024-03-25T19:43:01.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.758+0000 7f22897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:01.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.758+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2284003970 con 0x7f228c14be90 2024-03-25T19:43:01.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.758+0000 7f22897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.758+0000 7f22897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.758+0000 7f22897fa640 10 monclient: dump: 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:01.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.758+0000 7f228affd640 4 set_mon_vals no callback set 2024-03-25T19:43:01.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.759+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f2284019980 con 0x7f228c14be90 2024-03-25T19:43:01.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.760+0000 7f2290e64640 10 monclient: get_auth_request con 0x7f227006e0f0 auth_method 0 2024-03-25T19:43:01.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.760+0000 7f22897fa640 10 client.4550.objecter ms_handle_connect 0x7f227006e0f0 2024-03-25T19:43:01.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.765+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f2284053550 con 0x7f228c14be90 2024-03-25T19:43:01.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.765+0000 7f22897fa640 10 client.4550.objecter ms_dispatch 0x7f228c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:01.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.765+0000 7f22897fa640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:01.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.765+0000 7f22897fa640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:01.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.765+0000 7f22897fa640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-25T19:43:01.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.765+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f228404f060 con 0x7f228c14be90 2024-03-25T19:43:01.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.765+0000 7f22897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:01.766 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:01.765+0000 7f22897fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:02.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.010+0000 7f22930c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:02.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.010+0000 7f22930c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:02.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.010+0000 7f22930c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:02.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.010+0000 7f22930c8640 1 -- 172.21.15.107:0/4159667947 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f228c0627b0 con 0x7f228c14be90 2024-03-25T19:43:02.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.044+0000 7f22897fa640 1 -- 172.21.15.107:0/4159667947 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f2284050f90 con 0x7f228c14be90 2024-03-25T19:43:02.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.044+0000 7f22897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:02.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.044+0000 7f22897fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:02.045 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:43:02.045 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:02.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 10 librados: watch_flush enter 2024-03-25T19:43:02.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 10 librados: watch_flush exit 2024-03-25T19:43:02.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-25T19:43:02.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 10 client.4550.objecter successfully canceled tick 2024-03-25T19:43:02.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 1 -- 172.21.15.107:0/4159667947 >> v1:172.21.15.195:6800/2984408421 conn(0x7f227006e0f0 legacy=0x7f22700705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:02.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 10 monclient: shutdown 2024-03-25T19:43:02.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:02.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 1 -- 172.21.15.107:0/4159667947 >> v1:172.21.15.107:6790/0 conn(0x7f228c14be90 legacy=0x7f228c1e2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:02.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 1 -- 172.21.15.107:0/4159667947 shutdown_connections 2024-03-25T19:43:02.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 1 -- 172.21.15.107:0/4159667947 >> 172.21.15.107:0/4159667947 conn(0x7f228c0617a0 msgr2=0x7f228c137f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:02.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 1 -- 172.21.15.107:0/4159667947 shutdown_connections 2024-03-25T19:43:02.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 1 -- 172.21.15.107:0/4159667947 wait complete. 2024-03-25T19:43:02.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:02.047+0000 7f22930c8640 1 librados: shutdown 2024-03-25T19:43:02.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:05.610 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:07:35.926 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:05.623 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:07:35.939 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:05.649 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:07:35.966 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:05.666 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:07:35.952 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:05.666 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:07:35.982 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:05.671 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:07:35.987 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:05.674 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:07:35.960 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:05.701 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:07:35.988 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:05.705 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:07:36.022 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:08.866 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:08.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.978+0000 7f5a9e655640 10 monclient: get_monmap_and_config 2024-03-25T19:43:08.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.978+0000 7f5a9e655640 10 monclient: build_initial_monmap 2024-03-25T19:43:08.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.978+0000 7f5a9e655640 10 monclient: monmap: 2024-03-25T19:43:08.978 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:08.978 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:08.978 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:43:08.979593+0000 2024-03-25T19:43:08.978 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:43:08.979593+0000 2024-03-25T19:43:08.978 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:08.978 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:08.979 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:08.979 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:43:08.979 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:43:08.979 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:08.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.979+0000 7f5a9e655640 1 Processor -- start 2024-03-25T19:43:08.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.979+0000 7f5a9e655640 1 -- start start 2024-03-25T19:43:08.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.979+0000 7f5a9e655640 10 monclient: init 2024-03-25T19:43:08.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.980+0000 7f5a9e655640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:08.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.980+0000 7f5a9e655640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:08.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.980+0000 7f5a9e655640 10 monclient(hunting): picked mon.noname-c con 0x7f5a9813fc60 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:08.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.980+0000 7f5a9e655640 10 monclient(hunting): picked mon.noname-a con 0x7f5a98070570 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:08.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.980+0000 7f5a9e655640 10 monclient(hunting): picked mon.noname-b con 0x7f5a980727a0 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:08.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.980+0000 7f5a9e655640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9813c9f0 con 0x7f5a98070570 2024-03-25T19:43:08.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.980+0000 7f5a9e655640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9806fac0 con 0x7f5a9813fc60 2024-03-25T19:43:08.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.980+0000 7f5a9e655640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a980715d0 con 0x7f5a980727a0 2024-03-25T19:43:08.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.980+0000 7f5a9e655640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:08.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.980+0000 7f5a9e655640 10 monclient(hunting): authenticate will time out at 4981.321441s 2024-03-25T19:43:08.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.988+0000 7f5a97fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f5a9813fc60 0x7f5a9806f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:35898/0 (socket says 172.21.15.107:35898) 2024-03-25T19:43:08.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.988+0000 7f5a97fff640 1 -- 172.21.15.107:0/2675395300 learned_addr learned my addr 172.21.15.107:0/2675395300 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:08.999 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.999+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773099260 0 0) 0x7f5a9806fac0 con 0x7f5a9813fc60 2024-03-25T19:43:08.999 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.999+0000 7f5a96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:08.999 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.999+0000 7f5a96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:08.999 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.999+0000 7f5a96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:08.999 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.999+0000 7f5a96ffd640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:43:08.999 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:08.999+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a80003520 con 0x7f5a9813fc60 2024-03-25T19:43:09.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.003+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538675457 0 0) 0x7f5a9813c9f0 con 0x7f5a98070570 2024-03-25T19:43:09.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.003+0000 7f5a96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:09.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.003+0000 7f5a96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:09.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.003+0000 7f5a96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:09.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.003+0000 7f5a96ffd640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:43:09.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.003+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a9806fac0 con 0x7f5a98070570 2024-03-25T19:43:09.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.006+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900400079 0 0) 0x7f5a980715d0 con 0x7f5a980727a0 2024-03-25T19:43:09.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.006+0000 7f5a96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:09.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.006+0000 7f5a96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:09.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.006+0000 7f5a96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:09.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.006+0000 7f5a96ffd640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:43:09.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.006+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a9813c9f0 con 0x7f5a980727a0 2024-03-25T19:43:09.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1223244836 0 0) 0x7f5a80003520 con 0x7f5a9813fc60 2024-03-25T19:43:09.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 >> v1:172.21.15.107:6790/0 conn(0x7f5a980727a0 legacy=0x7f5a9814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 >> v1:172.21.15.107:6789/0 conn(0x7f5a98070570 legacy=0x7f5a98072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 1 monclient: found mon.noname-c 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a980718b0 con 0x7f5a9813fc60 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5a88001690 con 0x7f5a9813fc60 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.015+0000 7f5a96ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.016+0000 7f5a96ffd640 10 monclient: dump: 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:09.016 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:09.017 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:09.017 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:09.017 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:09.017 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:09.017 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:09.017 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:09.017 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:09.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.016+0000 7f5a9e655640 5 monclient: authenticate success, global_id 4540 2024-03-25T19:43:09.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.016+0000 7f5a9e655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:09.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.033+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a880017f0 con 0x7f5a9813fc60 2024-03-25T19:43:09.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.033+0000 7f5a96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:09.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.033+0000 7f5a96ffd640 1 -- 172.21.15.107:0/2675395300 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5a88001c00 con 0x7f5a9813fc60 2024-03-25T19:43:09.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.033+0000 7f5a9e655640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:09.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.033+0000 7f5a9e655640 4 set_mon_vals no callback set 2024-03-25T19:43:09.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.033+0000 7f5a96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:09.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.033+0000 7f5a96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:09.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.033+0000 7f5a96ffd640 10 monclient: dump: 2024-03-25T19:43:09.033 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:09.033 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:09.034 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:09.034 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:09.034 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:09.034 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:09.034 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:09.034 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:09.034 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:09.034 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.034+0000 7f5a9e655640 10 monclient: shutdown 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.034+0000 7f5a9e655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.034+0000 7f5a9e655640 1 -- 172.21.15.107:0/2675395300 >> v1:172.21.15.195:6789/0 conn(0x7f5a9813fc60 legacy=0x7f5a9806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.034+0000 7f5a9e655640 1 -- 172.21.15.107:0/2675395300 shutdown_connections 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.034+0000 7f5a9e655640 1 -- 172.21.15.107:0/2675395300 >> 172.21.15.107:0/2675395300 conn(0x7f5a980b4a80 msgr2=0x7f5a980b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.034+0000 7f5a9e655640 1 -- 172.21.15.107:0/2675395300 shutdown_connections 2024-03-25T19:43:09.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.035+0000 7f5a9e655640 1 -- 172.21.15.107:0/2675395300 wait complete. 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.036+0000 7f5a9e655640 10 monclient: build_initial_monmap 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.036+0000 7f5a9e655640 10 monclient: monmap: 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:43:09.036 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.036+0000 7f5a9e655640 1 librados: starting msgr at 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.036+0000 7f5a9e655640 1 librados: starting objecter 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.036+0000 7f5a9e655640 1 Processor -- start 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.036+0000 7f5a9e655640 1 -- start start 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.036+0000 7f5a9e655640 1 librados: setting wanted keys 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.036+0000 7f5a9e655640 1 librados: calling monclient init 2024-03-25T19:43:09.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.036+0000 7f5a9e655640 10 monclient: init 2024-03-25T19:43:09.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.037+0000 7f5a9e655640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:09.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.037+0000 7f5a9e655640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:09.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.037+0000 7f5a9e655640 10 monclient(hunting): picked mon.noname-c con 0x7f5a98070570 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:09.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.037+0000 7f5a9e655640 10 monclient(hunting): picked mon.noname-b con 0x7f5a980727a0 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:09.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.037+0000 7f5a9e655640 10 monclient(hunting): picked mon.noname-a con 0x7f5a9813fc60 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:09.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.037+0000 7f5a9e655640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a98135690 con 0x7f5a9813fc60 2024-03-25T19:43:09.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.037+0000 7f5a9e655640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a98143320 con 0x7f5a980727a0 2024-03-25T19:43:09.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.037+0000 7f5a9e655640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a98142990 con 0x7f5a98070570 2024-03-25T19:43:09.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.037+0000 7f5a9e655640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:09.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.037+0000 7f5a9e655640 10 monclient(hunting): authenticate will time out at 4981.378435s 2024-03-25T19:43:09.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.042+0000 7f5a97fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f5a98070570 0x7f5a981d3600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42268/0 (socket says 172.21.15.107:42268) 2024-03-25T19:43:09.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.042+0000 7f5a97fff640 1 -- 172.21.15.107:0/213726935 learned_addr learned my addr 172.21.15.107:0/213726935 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:09.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.044+0000 7f5a94ff9640 10 client.?.objecter ms_handle_connect 0x7f5a98070570 2024-03-25T19:43:09.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.044+0000 7f5a94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:09.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.044+0000 7f5a94ff9640 10 client.?.objecter ms_handle_connect 0x7f5a980727a0 2024-03-25T19:43:09.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.044+0000 7f5a94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:09.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.053+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151263498 0 0) 0x7f5a98142990 con 0x7f5a98070570 2024-03-25T19:43:09.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.054+0000 7f5a94ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:09.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.054+0000 7f5a94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:09.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.054+0000 7f5a94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:09.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.054+0000 7f5a94ff9640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:43:09.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.054+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a84004eb0 con 0x7f5a98070570 2024-03-25T19:43:09.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.057+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549412483 0 0) 0x7f5a98143320 con 0x7f5a980727a0 2024-03-25T19:43:09.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.057+0000 7f5a94ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:09.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.057+0000 7f5a94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:09.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.057+0000 7f5a94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:09.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.057+0000 7f5a94ff9640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:43:09.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.057+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a98142990 con 0x7f5a980727a0 2024-03-25T19:43:09.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.059+0000 7f5a94ff9640 10 client.?.objecter ms_handle_connect 0x7f5a9813fc60 2024-03-25T19:43:09.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.059+0000 7f5a94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:09.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.072+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619378954 0 0) 0x7f5a98135690 con 0x7f5a9813fc60 2024-03-25T19:43:09.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.072+0000 7f5a94ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:09.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.072+0000 7f5a94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:09.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.072+0000 7f5a94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:09.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.072+0000 7f5a94ff9640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:43:09.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.072+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a98143320 con 0x7f5a9813fc60 2024-03-25T19:43:09.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.076+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194901406 0 0) 0x7f5a98142990 con 0x7f5a980727a0 2024-03-25T19:43:09.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.077+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a98135690 con 0x7f5a980727a0 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.077+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5a8c001970 con 0x7f5a980727a0 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.077+0000 7f5a94ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.077+0000 7f5a94ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.077+0000 7f5a94ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:09.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.077+0000 7f5a94ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.089+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 26328730 0 0) 0x7f5a84004eb0 con 0x7f5a98070570 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.089+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a98142990 con 0x7f5a98070570 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.089+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5a88001bc0 con 0x7f5a98070570 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.089+0000 7f5a94ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.089+0000 7f5a94ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.089+0000 7f5a94ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:09.089 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:09.090 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:09.090 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:09.090 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:09.091 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:09.091 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:09.091 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:09.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2097856165 0 0) 0x7f5a98142990 con 0x7f5a98070570 2024-03-25T19:43:09.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 >> v1:172.21.15.195:6789/0 conn(0x7f5a980727a0 legacy=0x7f5a98134870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 >> v1:172.21.15.107:6789/0 conn(0x7f5a9813fc60 legacy=0x7f5a98134f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a94ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:09.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a94ff9640 1 monclient: found mon.c 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a94ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a94ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a98142b20 con 0x7f5a98070570 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a94ff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a94ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a9e655640 5 monclient: authenticate success, global_id 4556 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a9e655640 10 monclient: _renew_subs 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a9e655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a9e655640 1 -- 172.21.15.107:0/213726935 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a98141560 con 0x7f5a98070570 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a9e655640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a9e655640 10 monclient: _renew_subs 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a9e655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a9e655640 1 -- 172.21.15.107:0/213726935 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a981d3ef0 con 0x7f5a98070570 2024-03-25T19:43:09.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.113+0000 7f5a9e655640 1 librados: init done 2024-03-25T19:43:09.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.114+0000 7f5a9e655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:09.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.114+0000 7f5a9e655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:09.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.114+0000 7f5a9e655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:09.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.114+0000 7f5a9e655640 1 -- 172.21.15.107:0/213726935 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a980627b0 con 0x7f5a98070570 2024-03-25T19:43:09.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.376+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a880011f0 con 0x7f5a98070570 2024-03-25T19:43:09.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.376+0000 7f5a94ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:09.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.376+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5a88004c30 con 0x7f5a98070570 2024-03-25T19:43:09.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.376+0000 7f5a94ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:09.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.376+0000 7f5a94ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:09.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.376+0000 7f5a94ff9640 10 monclient: dump: 2024-03-25T19:43:09.376 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:09.376 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:09.376 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:09.376 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:09.376 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:09.376 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:09.377 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:09.377 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:09.377 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:09.377 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:09.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.376+0000 7f5a967fc640 4 set_mon_vals no callback set 2024-03-25T19:43:09.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.377+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f5a8801ac40 con 0x7f5a98070570 2024-03-25T19:43:09.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.377+0000 7f5a977fe640 10 monclient: get_auth_request con 0x7f5a8406de70 auth_method 0 2024-03-25T19:43:09.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.377+0000 7f5a94ff9640 10 client.4556.objecter ms_handle_connect 0x7f5a8406de70 2024-03-25T19:43:09.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.382+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f5a88054930 con 0x7f5a98070570 2024-03-25T19:43:09.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.382+0000 7f5a94ff9640 10 client.4556.objecter ms_dispatch 0x7f5a98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:09.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.382+0000 7f5a94ff9640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:09.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.382+0000 7f5a94ff9640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:09.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.382+0000 7f5a94ff9640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-25T19:43:09.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.382+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a88050270 con 0x7f5a98070570 2024-03-25T19:43:09.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.382+0000 7f5a94ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:09.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.382+0000 7f5a94ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:09.619 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.619+0000 7f5a9e655640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:09.619 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.619+0000 7f5a9e655640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:09.619 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.619+0000 7f5a9e655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:09.619 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.619+0000 7f5a9e655640 1 -- 172.21.15.107:0/213726935 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a9806af80 con 0x7f5a98070570 2024-03-25T19:43:09.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.650+0000 7f5a94ff9640 1 -- 172.21.15.107:0/213726935 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f5a88052370 con 0x7f5a98070570 2024-03-25T19:43:09.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.650+0000 7f5a94ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:09.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.650+0000 7f5a94ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:09.651 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:43:09.651 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:09.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 10 librados: watch_flush enter 2024-03-25T19:43:09.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 10 librados: watch_flush exit 2024-03-25T19:43:09.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-25T19:43:09.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 10 client.4556.objecter successfully canceled tick 2024-03-25T19:43:09.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 1 -- 172.21.15.107:0/213726935 >> v1:172.21.15.195:6800/2984408421 conn(0x7f5a8406de70 legacy=0x7f5a84070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 10 monclient: shutdown 2024-03-25T19:43:09.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:09.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 1 -- 172.21.15.107:0/213726935 >> v1:172.21.15.107:6790/0 conn(0x7f5a98070570 legacy=0x7f5a981d3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:09.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 1 -- 172.21.15.107:0/213726935 shutdown_connections 2024-03-25T19:43:09.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 1 -- 172.21.15.107:0/213726935 >> 172.21.15.107:0/213726935 conn(0x7f5a980b4a80 msgr2=0x7f5a980afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:09.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 1 -- 172.21.15.107:0/213726935 shutdown_connections 2024-03-25T19:43:09.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 1 -- 172.21.15.107:0/213726935 wait complete. 2024-03-25T19:43:09.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:09.653+0000 7f5a9e655640 1 librados: shutdown 2024-03-25T19:43:09.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:16.472 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:16.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.590+0000 7f50b6665640 10 monclient: get_monmap_and_config 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.590+0000 7f50b6665640 10 monclient: build_initial_monmap 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.590+0000 7f50b6665640 10 monclient: monmap: 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:43:16.592129+0000 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:43:16.592129+0000 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:43:16.591 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:16.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.591+0000 7f50b6665640 1 Processor -- start 2024-03-25T19:43:16.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.591+0000 7f50b6665640 1 -- start start 2024-03-25T19:43:16.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.591+0000 7f50b6665640 10 monclient: init 2024-03-25T19:43:16.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.592+0000 7f50b6665640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:16.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.592+0000 7f50b6665640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:16.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.592+0000 7f50b6665640 10 monclient(hunting): picked mon.noname-a con 0x7f50b0061830 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:16.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.592+0000 7f50b6665640 10 monclient(hunting): picked mon.noname-c con 0x7f50b00ae840 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:16.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.592+0000 7f50b6665640 10 monclient(hunting): picked mon.noname-b con 0x7f50b013fbd0 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:16.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.592+0000 7f50b6665640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b00ae1c0 con 0x7f50b0061830 2024-03-25T19:43:16.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.592+0000 7f50b6665640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b006f7d0 con 0x7f50b00ae840 2024-03-25T19:43:16.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.592+0000 7f50b6665640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b006fa40 con 0x7f50b013fbd0 2024-03-25T19:43:16.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.592+0000 7f50b6665640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:16.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.592+0000 7f50b6665640 10 monclient(hunting): authenticate will time out at 4988.934265s 2024-03-25T19:43:16.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.599+0000 7f50affff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f50b0061830 0x7f50b013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40576/0 (socket says 172.21.15.107:40576) 2024-03-25T19:43:16.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.599+0000 7f50affff640 1 -- 172.21.15.107:0/2727599291 learned_addr learned my addr 172.21.15.107:0/2727599291 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:16.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.611+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188539522 0 0) 0x7f50b00ae1c0 con 0x7f50b0061830 2024-03-25T19:43:16.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.611+0000 7f50aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:16.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.611+0000 7f50aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:16.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.611+0000 7f50aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:16.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.611+0000 7f50aeffd640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:43:16.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.611+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5098003520 con 0x7f50b0061830 2024-03-25T19:43:16.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.615+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000197846 0 0) 0x7f50b006fa40 con 0x7f50b013fbd0 2024-03-25T19:43:16.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.615+0000 7f50aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:16.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.615+0000 7f50aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:16.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.615+0000 7f50aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:16.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.615+0000 7f50aeffd640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:43:16.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.615+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b00ae1c0 con 0x7f50b013fbd0 2024-03-25T19:43:16.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.616+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212407535 0 0) 0x7f50b006f7d0 con 0x7f50b00ae840 2024-03-25T19:43:16.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.616+0000 7f50aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:16.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.616+0000 7f50aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:16.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.616+0000 7f50aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:16.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.616+0000 7f50aeffd640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:43:16.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.616+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b006fa40 con 0x7f50b00ae840 2024-03-25T19:43:16.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1787979832 0 0) 0x7f5098003520 con 0x7f50b0061830 2024-03-25T19:43:16.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 >> v1:172.21.15.107:6790/0 conn(0x7f50b013fbd0 legacy=0x7f50b0145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 >> v1:172.21.15.195:6789/0 conn(0x7f50b00ae840 legacy=0x7f50b0145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:16.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 1 monclient: found mon.noname-a 2024-03-25T19:43:16.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:16.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0073040 con 0x7f50b0061830 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f50a0001690 con 0x7f50b0061830 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50aeffd640 10 monclient: dump: 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:16.624 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:16.625 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:16.625 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:16.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50b6665640 5 monclient: authenticate success, global_id 4554 2024-03-25T19:43:16.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.622+0000 7f50b6665640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:16.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.631+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50a00017f0 con 0x7f50b0061830 2024-03-25T19:43:16.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.631+0000 7f50aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:16.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.631+0000 7f50aeffd640 1 -- 172.21.15.107:0/2727599291 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f50a0001b80 con 0x7f50b0061830 2024-03-25T19:43:16.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.631+0000 7f50aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:16.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.631+0000 7f50aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.631+0000 7f50aeffd640 10 monclient: dump: 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.631+0000 7f50b6665640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:16.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.631+0000 7f50b6665640 4 set_mon_vals no callback set 2024-03-25T19:43:16.635 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.632+0000 7f50b6665640 10 monclient: shutdown 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.632+0000 7f50b6665640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.632+0000 7f50b6665640 1 -- 172.21.15.107:0/2727599291 >> v1:172.21.15.107:6789/0 conn(0x7f50b0061830 legacy=0x7f50b013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.632+0000 7f50b6665640 1 -- 172.21.15.107:0/2727599291 shutdown_connections 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.632+0000 7f50b6665640 1 -- 172.21.15.107:0/2727599291 >> 172.21.15.107:0/2727599291 conn(0x7f50b00b0ef0 msgr2=0x7f50b00b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.632+0000 7f50b6665640 1 -- 172.21.15.107:0/2727599291 shutdown_connections 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.632+0000 7f50b6665640 1 -- 172.21.15.107:0/2727599291 wait complete. 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.633+0000 7f50b6665640 10 monclient: build_initial_monmap 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.633+0000 7f50b6665640 10 monclient: monmap: 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:43:16.636 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.633+0000 7f50b6665640 1 librados: starting msgr at 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.633+0000 7f50b6665640 1 librados: starting objecter 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 1 Processor -- start 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 1 -- start start 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 1 librados: setting wanted keys 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 1 librados: calling monclient init 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 10 monclient: init 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 10 monclient(hunting): picked mon.noname-c con 0x7f50b0061830 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 10 monclient(hunting): picked mon.noname-a con 0x7f50b00ae840 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 10 monclient(hunting): picked mon.noname-b con 0x7f50b013fbd0 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:16.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b013d8a0 con 0x7f50b00ae840 2024-03-25T19:43:16.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b013b020 con 0x7f50b013fbd0 2024-03-25T19:43:16.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b01393b0 con 0x7f50b0061830 2024-03-25T19:43:16.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:16.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.634+0000 7f50b6665640 10 monclient(hunting): authenticate will time out at 4988.976679s 2024-03-25T19:43:16.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.640+0000 7f50affff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f50b0061830 0x7f50b01372d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34140/0 (socket says 172.21.15.107:34140) 2024-03-25T19:43:16.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.640+0000 7f50affff640 1 -- 172.21.15.107:0/107710524 learned_addr learned my addr 172.21.15.107:0/107710524 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:16.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.640+0000 7f50acff9640 10 client.?.objecter ms_handle_connect 0x7f50b0061830 2024-03-25T19:43:16.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.640+0000 7f50acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:16.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.644+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309382079 0 0) 0x7f50b01393b0 con 0x7f50b0061830 2024-03-25T19:43:16.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.644+0000 7f50acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:16.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.644+0000 7f50acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:16.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.644+0000 7f50acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:16.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.644+0000 7f50acff9640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:43:16.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.644+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f509c004eb0 con 0x7f50b0061830 2024-03-25T19:43:16.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.645+0000 7f50acff9640 10 client.?.objecter ms_handle_connect 0x7f50b00ae840 2024-03-25T19:43:16.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.645+0000 7f50acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:16.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.646+0000 7f50acff9640 10 client.?.objecter ms_handle_connect 0x7f50b013fbd0 2024-03-25T19:43:16.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.646+0000 7f50acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:16.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.654+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465934836 0 0) 0x7f50b013d8a0 con 0x7f50b00ae840 2024-03-25T19:43:16.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.654+0000 7f50acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:16.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.654+0000 7f50acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:16.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.654+0000 7f50acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:16.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.654+0000 7f50acff9640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:43:16.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.654+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b01393b0 con 0x7f50b00ae840 2024-03-25T19:43:16.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.657+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214223092 0 0) 0x7f50b013b020 con 0x7f50b013fbd0 2024-03-25T19:43:16.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.657+0000 7f50acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:16.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.657+0000 7f50acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:16.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.657+0000 7f50acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:16.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.657+0000 7f50acff9640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:43:16.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.657+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b013d8a0 con 0x7f50b013fbd0 2024-03-25T19:43:16.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.672+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3064484984 0 0) 0x7f509c004eb0 con 0x7f50b0061830 2024-03-25T19:43:16.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.672+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50b013b020 con 0x7f50b0061830 2024-03-25T19:43:16.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.672+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f50a0001b40 con 0x7f50b0061830 2024-03-25T19:43:16.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.672+0000 7f50acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:16.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.672+0000 7f50acff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:16.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.672+0000 7f50acff9640 10 monclient(hunting): dump: 2024-03-25T19:43:16.673 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:16.673 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:16.674 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:16.674 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:16.674 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:16.674 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:16.674 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:16.674 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:16.674 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:16.674 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:16.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.672+0000 7f50acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:16.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.675+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 482490103 0 0) 0x7f50b013d8a0 con 0x7f50b013fbd0 2024-03-25T19:43:16.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.675+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f509c004eb0 con 0x7f50b013fbd0 2024-03-25T19:43:16.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.675+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f50a8001cb0 con 0x7f50b013fbd0 2024-03-25T19:43:16.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.675+0000 7f50acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:16.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.675+0000 7f50acff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:16.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.675+0000 7f50acff9640 10 monclient(hunting): dump: 2024-03-25T19:43:16.676 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:16.676 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:16.676 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:16.676 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:16.677 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:16.677 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:16.677 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:16.677 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:16.677 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:16.677 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:16.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2871587003 0 0) 0x7f50b013b020 con 0x7f50b0061830 2024-03-25T19:43:16.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 >> v1:172.21.15.195:6789/0 conn(0x7f50b013fbd0 legacy=0x7f50b01e2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 >> v1:172.21.15.107:6789/0 conn(0x7f50b00ae840 legacy=0x7f50b01379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50acff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:16.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50acff9640 1 monclient: found mon.c 2024-03-25T19:43:16.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:16.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50acff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:16.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0139540 con 0x7f50b0061830 2024-03-25T19:43:16.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50acff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:16.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50acff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:16.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:16.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50b6665640 5 monclient: authenticate success, global_id 4562 2024-03-25T19:43:16.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50b6665640 10 monclient: _renew_subs 2024-03-25T19:43:16.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50b6665640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:16.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50b6665640 1 -- 172.21.15.107:0/107710524 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b0139c30 con 0x7f50b0061830 2024-03-25T19:43:16.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50b6665640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:16.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50b6665640 10 monclient: _renew_subs 2024-03-25T19:43:16.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50b6665640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:16.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50b6665640 1 -- 172.21.15.107:0/107710524 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b013aa90 con 0x7f50b0061830 2024-03-25T19:43:16.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.678+0000 7f50b6665640 1 librados: init done 2024-03-25T19:43:16.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.679+0000 7f50b6665640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:16.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.679+0000 7f50b6665640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:16.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.679+0000 7f50b6665640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:16.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.679+0000 7f50b6665640 1 -- 172.21.15.107:0/107710524 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b013b1b0 con 0x7f50b0061830 2024-03-25T19:43:16.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.997+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50a00011f0 con 0x7f50b0061830 2024-03-25T19:43:16.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.997+0000 7f50acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:16.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.997+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f50a0001d80 con 0x7f50b0061830 2024-03-25T19:43:16.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.997+0000 7f50acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:16.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.997+0000 7f50acff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:16.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.997+0000 7f50acff9640 10 monclient: dump: 2024-03-25T19:43:16.998 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:16.998 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:16.998 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:16.998 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:16.998 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:16.999 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:16.999 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:16.999 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:16.999 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:16.999 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:16.999 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.997+0000 7f50ae7fc640 4 set_mon_vals no callback set 2024-03-25T19:43:16.999 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.997+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f50a001ab20 con 0x7f50b0061830 2024-03-25T19:43:17.000 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.998+0000 7f50af7fe640 10 monclient: get_auth_request con 0x7f509c06df20 auth_method 0 2024-03-25T19:43:17.000 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:16.998+0000 7f50acff9640 10 client.4562.objecter ms_handle_connect 0x7f509c06df20 2024-03-25T19:43:17.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.003+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f50a0054830 con 0x7f50b0061830 2024-03-25T19:43:17.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.003+0000 7f50acff9640 10 client.4562.objecter ms_dispatch 0x7f50b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:17.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.003+0000 7f50acff9640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:17.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.003+0000 7f50acff9640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:17.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.003+0000 7f50acff9640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-25T19:43:17.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.003+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50a0050170 con 0x7f50b0061830 2024-03-25T19:43:17.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.003+0000 7f50acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:17.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.003+0000 7f50acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:17.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.245+0000 7f50b6665640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:17.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.245+0000 7f50b6665640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:17.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.245+0000 7f50b6665640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:17.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.245+0000 7f50b6665640 1 -- 172.21.15.107:0/107710524 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50b00627b0 con 0x7f50b0061830 2024-03-25T19:43:17.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.336+0000 7f50acff9640 1 -- 172.21.15.107:0/107710524 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f50a0052270 con 0x7f50b0061830 2024-03-25T19:43:17.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.336+0000 7f50acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:17.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.336+0000 7f50acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:17.337 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:43:17.337 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:17.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 10 librados: watch_flush enter 2024-03-25T19:43:17.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 10 librados: watch_flush exit 2024-03-25T19:43:17.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-25T19:43:17.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 10 client.4562.objecter successfully canceled tick 2024-03-25T19:43:17.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 1 -- 172.21.15.107:0/107710524 >> v1:172.21.15.195:6800/2984408421 conn(0x7f509c06df20 legacy=0x7f509c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:17.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 10 monclient: shutdown 2024-03-25T19:43:17.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:17.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 1 -- 172.21.15.107:0/107710524 >> v1:172.21.15.107:6790/0 conn(0x7f50b0061830 legacy=0x7f50b01372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:17.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 1 -- 172.21.15.107:0/107710524 shutdown_connections 2024-03-25T19:43:17.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 1 -- 172.21.15.107:0/107710524 >> 172.21.15.107:0/107710524 conn(0x7f50b00b0ef0 msgr2=0x7f50b013f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:17.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 1 -- 172.21.15.107:0/107710524 shutdown_connections 2024-03-25T19:43:17.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 1 -- 172.21.15.107:0/107710524 wait complete. 2024-03-25T19:43:17.340 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:17.338+0000 7f50b6665640 1 librados: shutdown 2024-03-25T19:43:17.352 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:24.158 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.277+0000 7f6319081640 10 monclient: get_monmap_and_config 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.277+0000 7f6319081640 10 monclient: build_initial_monmap 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.277+0000 7f6319081640 10 monclient: monmap: 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:43:24.278458+0000 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:43:24.278458+0000 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:43:24.277 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.278+0000 7f6319081640 1 Processor -- start 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.278+0000 7f6319081640 1 -- start start 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.278+0000 7f6319081640 10 monclient: init 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.278+0000 7f6319081640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.278+0000 7f6319081640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.278+0000 7f6319081640 10 monclient(hunting): picked mon.noname-b con 0x7f63140a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.278+0000 7f6319081640 10 monclient(hunting): picked mon.noname-a con 0x7f631413f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.279+0000 7f6319081640 10 monclient(hunting): picked mon.noname-c con 0x7f63141430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.279+0000 7f6319081640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6314142520 con 0x7f631413f710 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.279+0000 7f6319081640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63140b28a0 con 0x7f63141430c0 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.279+0000 7f6319081640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63140b2a30 con 0x7f63140a98f0 2024-03-25T19:43:24.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.279+0000 7f6319081640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:24.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.279+0000 7f6319081640 10 monclient(hunting): authenticate will time out at 4996.619963s 2024-03-25T19:43:24.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.280+0000 7f6312d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f63140a98f0 0x7f63140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34462/0 (socket says 172.21.15.107:34462) 2024-03-25T19:43:24.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.280+0000 7f6312d9d640 1 -- 172.21.15.107:0/1053017326 learned_addr learned my addr 172.21.15.107:0/1053017326 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:24.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.295+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938050731 0 0) 0x7f63140b2a30 con 0x7f63140a98f0 2024-03-25T19:43:24.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.295+0000 7f6311d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:24.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.295+0000 7f6311d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:24.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.295+0000 7f6311d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:24.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.295+0000 7f6311d9b640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:43:24.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.295+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62fc003520 con 0x7f63140a98f0 2024-03-25T19:43:24.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.302+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969543245 0 0) 0x7f63140b28a0 con 0x7f63141430c0 2024-03-25T19:43:24.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.303+0000 7f6311d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:24.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.303+0000 7f6311d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:24.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.303+0000 7f6311d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:24.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.303+0000 7f6311d9b640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:43:24.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.303+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63140b2a30 con 0x7f63141430c0 2024-03-25T19:43:24.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351931960 0 0) 0x7f6314142520 con 0x7f631413f710 2024-03-25T19:43:24.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:24.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:24.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:24.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:43:24.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63140b28a0 con 0x7f631413f710 2024-03-25T19:43:24.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 319359807 0 0) 0x7f62fc003520 con 0x7f63140a98f0 2024-03-25T19:43:24.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 >> v1:172.21.15.195:6789/0 conn(0x7f63141430c0 legacy=0x7f631414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.312 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 >> v1:172.21.15.107:6789/0 conn(0x7f631413f710 legacy=0x7f63141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 1 monclient: found mon.noname-b 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63140b44c0 con 0x7f63140a98f0 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 10 monclient: _finish_auth 0 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6304001690 con 0x7f63140a98f0 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6319081640 5 monclient: authenticate success, global_id 4565 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6319081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6311d9b640 10 monclient: dump: 2024-03-25T19:43:24.313 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:24.314 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:24.314 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.314 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.314 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:24.314 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:24.314 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:24.314 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:24.314 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:24.314 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:24.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.312+0000 7f6319081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:24.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63040017f0 con 0x7f63140a98f0 2024-03-25T19:43:24.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6311d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6311d9b640 1 -- 172.21.15.107:0/1053017326 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6304001c00 con 0x7f63140a98f0 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6319081640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6319081640 4 set_mon_vals no callback set 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6311d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6311d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6311d9b640 10 monclient: dump: 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:24.326 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:24.327 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:24.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6319081640 10 monclient: shutdown 2024-03-25T19:43:24.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6319081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:24.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6319081640 1 -- 172.21.15.107:0/1053017326 >> v1:172.21.15.107:6790/0 conn(0x7f63140a98f0 legacy=0x7f63140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6319081640 1 -- 172.21.15.107:0/1053017326 shutdown_connections 2024-03-25T19:43:24.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.326+0000 7f6319081640 1 -- 172.21.15.107:0/1053017326 >> 172.21.15.107:0/1053017326 conn(0x7f63140b7ea0 msgr2=0x7f63140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:24.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.327+0000 7f6319081640 1 -- 172.21.15.107:0/1053017326 shutdown_connections 2024-03-25T19:43:24.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.327+0000 7f6319081640 1 -- 172.21.15.107:0/1053017326 wait complete. 2024-03-25T19:43:24.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.327+0000 7f6319081640 10 monclient: build_initial_monmap 2024-03-25T19:43:24.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.327+0000 7f6319081640 10 monclient: monmap: 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.328+0000 7f6319081640 1 librados: starting msgr at 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.328+0000 7f6319081640 1 librados: starting objecter 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.328+0000 7f6319081640 1 Processor -- start 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.328+0000 7f6319081640 1 -- start start 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.328+0000 7f6319081640 1 librados: setting wanted keys 2024-03-25T19:43:24.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.328+0000 7f6319081640 1 librados: calling monclient init 2024-03-25T19:43:24.330 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.328+0000 7f6319081640 10 monclient: init 2024-03-25T19:43:24.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.329+0000 7f6319081640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:24.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.329+0000 7f6319081640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:24.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.329+0000 7f6319081640 10 monclient(hunting): picked mon.noname-c con 0x7f63140a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:24.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.329+0000 7f6319081640 10 monclient(hunting): picked mon.noname-a con 0x7f631413f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:24.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.329+0000 7f6319081640 10 monclient(hunting): picked mon.noname-b con 0x7f63141430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:24.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.329+0000 7f6319081640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63140abd30 con 0x7f631413f710 2024-03-25T19:43:24.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.329+0000 7f6319081640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63140abfa0 con 0x7f63141430c0 2024-03-25T19:43:24.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.329+0000 7f6319081640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63141d77a0 con 0x7f63140a98f0 2024-03-25T19:43:24.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.329+0000 7f6319081640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:24.331 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.329+0000 7f6319081640 10 monclient(hunting): authenticate will time out at 4996.670157s 2024-03-25T19:43:24.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.338+0000 7f631259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f631413f710 0x7f63141d4820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55294/0 (socket says 172.21.15.107:55294) 2024-03-25T19:43:24.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.338+0000 7f631259c640 1 -- 172.21.15.107:0/238705111 learned_addr learned my addr 172.21.15.107:0/238705111 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:24.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.339+0000 7f62fb7fe640 10 client.?.objecter ms_handle_connect 0x7f63140a98f0 2024-03-25T19:43:24.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.339+0000 7f62fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:24.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.346+0000 7f62fb7fe640 10 client.?.objecter ms_handle_connect 0x7f631413f710 2024-03-25T19:43:24.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.347+0000 7f62fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:24.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.350+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886890629 0 0) 0x7f63141d77a0 con 0x7f63140a98f0 2024-03-25T19:43:24.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.350+0000 7f62fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:24.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.350+0000 7f62fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:24.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.350+0000 7f62fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:24.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.350+0000 7f62fb7fe640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:43:24.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.350+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6300004eb0 con 0x7f63140a98f0 2024-03-25T19:43:24.352 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.352+0000 7f62fb7fe640 10 client.?.objecter ms_handle_connect 0x7f63141430c0 2024-03-25T19:43:24.352 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.352+0000 7f62fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:24.352 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.353+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230191253 0 0) 0x7f63140abd30 con 0x7f631413f710 2024-03-25T19:43:24.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.353+0000 7f62fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:24.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.353+0000 7f62fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:24.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.353+0000 7f62fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:24.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.353+0000 7f62fb7fe640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:43:24.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.353+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63141d77a0 con 0x7f631413f710 2024-03-25T19:43:24.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.359+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7175078 0 0) 0x7f63140abfa0 con 0x7f63141430c0 2024-03-25T19:43:24.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.359+0000 7f62fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:24.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.359+0000 7f62fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:24.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.359+0000 7f62fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:24.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.359+0000 7f62fb7fe640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:43:24.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.359+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63140abd30 con 0x7f63141430c0 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.371+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 419748503 0 0) 0x7f63140abd30 con 0x7f63141430c0 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.371+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63140abfa0 con 0x7f63141430c0 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.371+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f630c001bd0 con 0x7f63141430c0 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.371+0000 7f62fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.371+0000 7f62fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.371+0000 7f62fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:24.371 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:24.372 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:24.372 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:24.372 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:24.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.371+0000 7f62fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:24.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.372+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 190985252 0 0) 0x7f63141d77a0 con 0x7f631413f710 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.372+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63140abd30 con 0x7f631413f710 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.372+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f63080019f0 con 0x7f631413f710 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.372+0000 7f62fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.372+0000 7f62fb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.372+0000 7f62fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:24.373 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:24.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.375+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3643351485 0 0) 0x7f6300004eb0 con 0x7f63140a98f0 2024-03-25T19:43:24.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.375+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63141d77a0 con 0x7f63140a98f0 2024-03-25T19:43:24.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.375+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6304001b00 con 0x7f63140a98f0 2024-03-25T19:43:24.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.375+0000 7f62fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:24.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.375+0000 7f62fb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:24.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.375+0000 7f62fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:43:24.375 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:24.375 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:24.375 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.375 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.376 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:24.376 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:24.376 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:24.376 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:24.376 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:24.376 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:24.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1909591636 0 0) 0x7f63141d77a0 con 0x7f63140a98f0 2024-03-25T19:43:24.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 >> v1:172.21.15.195:6789/0 conn(0x7f63141430c0 legacy=0x7f63141385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 >> v1:172.21.15.107:6789/0 conn(0x7f631413f710 legacy=0x7f63141d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f62fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f62fb7fe640 1 monclient: found mon.c 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f62fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f62fb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63141d83d0 con 0x7f63140a98f0 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f62fb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f62fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f62fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f6319081640 5 monclient: authenticate success, global_id 4568 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f6319081640 10 monclient: _renew_subs 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f6319081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f6319081640 1 -- 172.21.15.107:0/238705111 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63141d41e0 con 0x7f63140a98f0 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f6319081640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f6319081640 10 monclient: _renew_subs 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f6319081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f6319081640 1 -- 172.21.15.107:0/238705111 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63141d5110 con 0x7f63140a98f0 2024-03-25T19:43:24.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.380+0000 7f6319081640 1 librados: init done 2024-03-25T19:43:24.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.381+0000 7f6319081640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:24.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.381+0000 7f6319081640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:24.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.381+0000 7f6319081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:24.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.381+0000 7f6319081640 1 -- 172.21.15.107:0/238705111 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63141d7930 con 0x7f63140a98f0 2024-03-25T19:43:24.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.687+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6304001220 con 0x7f63140a98f0 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.688+0000 7f62fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.688+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6304004c30 con 0x7f63140a98f0 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.688+0000 7f62fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.688+0000 7f62fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.688+0000 7f62fb7fe640 10 monclient: dump: 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:24.688 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:24.689 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:24.689 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:24.689 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:24.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.688+0000 7f631159a640 4 set_mon_vals no callback set 2024-03-25T19:43:24.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.688+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f630401ac00 con 0x7f63140a98f0 2024-03-25T19:43:24.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.689+0000 7f631259c640 10 monclient: get_auth_request con 0x7f630006e160 auth_method 0 2024-03-25T19:43:24.690 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.689+0000 7f62fb7fe640 10 client.4568.objecter ms_handle_connect 0x7f630006e160 2024-03-25T19:43:24.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.694+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f6304054860 con 0x7f63140a98f0 2024-03-25T19:43:24.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.695+0000 7f62fb7fe640 10 client.4568.objecter ms_dispatch 0x7f6314000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:24.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.695+0000 7f62fb7fe640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:24.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.695+0000 7f62fb7fe640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:24.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.695+0000 7f62fb7fe640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-25T19:43:24.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.695+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63040501a0 con 0x7f63140a98f0 2024-03-25T19:43:24.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.695+0000 7f62fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:24.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.695+0000 7f62fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:24.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.947+0000 7f6319081640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:24.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.947+0000 7f6319081640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:24.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.947+0000 7f6319081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:24.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.947+0000 7f6319081640 1 -- 172.21.15.107:0/238705111 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63141d8560 con 0x7f63140a98f0 2024-03-25T19:43:24.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.970+0000 7f62fb7fe640 1 -- 172.21.15.107:0/238705111 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f63040522a0 con 0x7f63140a98f0 2024-03-25T19:43:24.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.970+0000 7f62fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:24.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.970+0000 7f62fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:24.970 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:43:24.970 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:24.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 10 librados: watch_flush enter 2024-03-25T19:43:24.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 10 librados: watch_flush exit 2024-03-25T19:43:24.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-25T19:43:24.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 10 client.4568.objecter successfully canceled tick 2024-03-25T19:43:24.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 1 -- 172.21.15.107:0/238705111 >> v1:172.21.15.195:6800/2984408421 conn(0x7f630006e160 legacy=0x7f6300070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 10 monclient: shutdown 2024-03-25T19:43:24.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:24.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 1 -- 172.21.15.107:0/238705111 >> v1:172.21.15.107:6790/0 conn(0x7f63140a98f0 legacy=0x7f63141d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:24.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 1 -- 172.21.15.107:0/238705111 shutdown_connections 2024-03-25T19:43:24.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 1 -- 172.21.15.107:0/238705111 >> 172.21.15.107:0/238705111 conn(0x7f63140b7ea0 msgr2=0x7f631413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:24.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 1 -- 172.21.15.107:0/238705111 shutdown_connections 2024-03-25T19:43:24.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.973+0000 7f6319081640 1 -- 172.21.15.107:0/238705111 wait complete. 2024-03-25T19:43:24.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:24.974+0000 7f6319081640 1 librados: shutdown 2024-03-25T19:43:24.985 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:25.618 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:07:55.934 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:25.667 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:07:55.984 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:25.669 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:07:55.985 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:25.700 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:07:56.017 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:25.721 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:07:56.008 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:25.727 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:07:56.043 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:25.727 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:07:56.014 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:25.756 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:07:56.043 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:25.763 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:07:56.080 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:31.793 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:31.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.909+0000 7f589af32640 10 monclient: get_monmap_and_config 2024-03-25T19:43:31.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.909+0000 7f589af32640 10 monclient: build_initial_monmap 2024-03-25T19:43:31.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.909+0000 7f589af32640 10 monclient: monmap: 2024-03-25T19:43:31.909 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:31.909 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:31.909 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:43:31.910621+0000 2024-03-25T19:43:31.909 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:43:31.910621+0000 2024-03-25T19:43:31.909 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:31.910 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:31.910 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:31.910 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:43:31.910 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:43:31.910 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:31.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.910+0000 7f589af32640 1 Processor -- start 2024-03-25T19:43:31.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.910+0000 7f589af32640 1 -- start start 2024-03-25T19:43:31.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.910+0000 7f589af32640 10 monclient: init 2024-03-25T19:43:31.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.911+0000 7f589af32640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:31.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.911+0000 7f589af32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:31.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.911+0000 7f589af32640 10 monclient(hunting): picked mon.noname-b con 0x7f58940b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:31.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.911+0000 7f589af32640 10 monclient(hunting): picked mon.noname-a con 0x7f58940b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:31.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.911+0000 7f589af32640 10 monclient(hunting): picked mon.noname-c con 0x7f589414be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:31.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.911+0000 7f589af32640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940af940 con 0x7f58940b1730 2024-03-25T19:43:31.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.911+0000 7f589af32640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940b0a70 con 0x7f589414be90 2024-03-25T19:43:31.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.911+0000 7f589af32640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940b0d50 con 0x7f58940b4d60 2024-03-25T19:43:31.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.911+0000 7f589af32640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:31.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.911+0000 7f589af32640 10 monclient(hunting): authenticate will time out at 5004.251918s 2024-03-25T19:43:31.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.922+0000 7f58994cf640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f589414be90 0x7f589414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:42308/0 (socket says 172.21.15.107:42308) 2024-03-25T19:43:31.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.922+0000 7f58994cf640 1 -- 172.21.15.107:0/653738277 learned_addr learned my addr 172.21.15.107:0/653738277 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:31.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.938+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307933459 0 0) 0x7f58940b0d50 con 0x7f58940b4d60 2024-03-25T19:43:31.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.938+0000 7f58937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:31.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.938+0000 7f58937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:31.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.938+0000 7f58937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:31.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.938+0000 7f58937fe640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:43:31.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.938+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5884003520 con 0x7f58940b4d60 2024-03-25T19:43:31.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.940+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971026847 0 0) 0x7f58940b0a70 con 0x7f589414be90 2024-03-25T19:43:31.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.940+0000 7f58937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:31.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.940+0000 7f58937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:31.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.940+0000 7f58937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:31.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.940+0000 7f58937fe640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:43:31.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.940+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58940b0d50 con 0x7f589414be90 2024-03-25T19:43:31.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.943+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185725159 0 0) 0x7f58940af940 con 0x7f58940b1730 2024-03-25T19:43:31.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.943+0000 7f58937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:31.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.943+0000 7f58937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:31.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.943+0000 7f58937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:31.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.943+0000 7f58937fe640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:43:31.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.943+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58940b0a70 con 0x7f58940b1730 2024-03-25T19:43:31.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 694169719 0 0) 0x7f58940b0a70 con 0x7f58940b1730 2024-03-25T19:43:31.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 >> v1:172.21.15.107:6790/0 conn(0x7f58940b4d60 legacy=0x7f589413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:31.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 >> v1:172.21.15.195:6789/0 conn(0x7f589414be90 legacy=0x7f589414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:31.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:31.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 1 monclient: found mon.noname-a 2024-03-25T19:43:31.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:31.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:43:31.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58940b1030 con 0x7f58940b1730 2024-03-25T19:43:31.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f58800015b0 con 0x7f58940b1730 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f589af32640 5 monclient: authenticate success, global_id 4566 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f589af32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f58937fe640 10 monclient: dump: 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:31.964 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:31.965 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:31.965 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:31.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.963+0000 7f589af32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:31.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.998+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5880001710 con 0x7f58940b1730 2024-03-25T19:43:31.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.998+0000 7f58937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:31.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.998+0000 7f58937fe640 1 -- 172.21.15.107:0/653738277 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5880001b60 con 0x7f58940b1730 2024-03-25T19:43:31.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.998+0000 7f589af32640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:31.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.998+0000 7f589af32640 4 set_mon_vals no callback set 2024-03-25T19:43:31.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.998+0000 7f58937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:31.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.998+0000 7f58937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:31.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.998+0000 7f58937fe640 10 monclient: dump: 2024-03-25T19:43:31.998 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:31.998 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:31.999 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:31.999 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:31.999 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:31.999 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:31.999 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:31.999 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:31.999 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:31.999 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:32.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.999+0000 7f589af32640 10 monclient: shutdown 2024-03-25T19:43:32.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.999+0000 7f589af32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.999+0000 7f589af32640 1 -- 172.21.15.107:0/653738277 >> v1:172.21.15.107:6789/0 conn(0x7f58940b1730 legacy=0x7f58940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.999+0000 7f589af32640 1 -- 172.21.15.107:0/653738277 shutdown_connections 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.999+0000 7f589af32640 1 -- 172.21.15.107:0/653738277 >> 172.21.15.107:0/653738277 conn(0x7f58940617a0 msgr2=0x7f5894061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.999+0000 7f589af32640 1 -- 172.21.15.107:0/653738277 shutdown_connections 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:31.999+0000 7f589af32640 1 -- 172.21.15.107:0/653738277 wait complete. 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.000+0000 7f589af32640 10 monclient: build_initial_monmap 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.000+0000 7f589af32640 10 monclient: monmap: 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:43:32.003 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.000+0000 7f589af32640 1 librados: starting msgr at 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.000+0000 7f589af32640 1 librados: starting objecter 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.000+0000 7f589af32640 1 Processor -- start 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.000+0000 7f589af32640 1 -- start start 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.000+0000 7f589af32640 1 librados: setting wanted keys 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.000+0000 7f589af32640 1 librados: calling monclient init 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.000+0000 7f589af32640 10 monclient: init 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.001+0000 7f589af32640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.001+0000 7f589af32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.001+0000 7f589af32640 10 monclient(hunting): picked mon.noname-a con 0x7f58940b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.001+0000 7f589af32640 10 monclient(hunting): picked mon.noname-b con 0x7f58940b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.001+0000 7f589af32640 10 monclient(hunting): picked mon.noname-c con 0x7f589414be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.001+0000 7f589af32640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589413bc10 con 0x7f58940b1730 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.001+0000 7f589af32640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58941d8550 con 0x7f58940b4d60 2024-03-25T19:43:32.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.001+0000 7f589af32640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58941d86e0 con 0x7f589414be90 2024-03-25T19:43:32.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.001+0000 7f589af32640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:32.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.001+0000 7f589af32640 10 monclient(hunting): authenticate will time out at 5004.342407s 2024-03-25T19:43:32.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.008+0000 7f5893fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f58940b4d60 0x7f58941d7e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:42382/0 (socket says 172.21.15.107:42382) 2024-03-25T19:43:32.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.008+0000 7f5893fff640 1 -- 172.21.15.107:0/352284511 learned_addr learned my addr 172.21.15.107:0/352284511 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:32.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.009+0000 7f58917fa640 10 client.?.objecter ms_handle_connect 0x7f58940b4d60 2024-03-25T19:43:32.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.009+0000 7f58917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:32.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.015+0000 7f58917fa640 10 client.?.objecter ms_handle_connect 0x7f589414be90 2024-03-25T19:43:32.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.015+0000 7f58917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:32.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.023+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055459589 0 0) 0x7f58941d8550 con 0x7f58940b4d60 2024-03-25T19:43:32.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.023+0000 7f58917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:32.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.023+0000 7f58917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:32.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.023+0000 7f58917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:32.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.023+0000 7f58917fa640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:43:32.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.023+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5878004eb0 con 0x7f58940b4d60 2024-03-25T19:43:32.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.027+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484729746 0 0) 0x7f58941d86e0 con 0x7f589414be90 2024-03-25T19:43:32.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.027+0000 7f58917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:32.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.027+0000 7f58917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:32.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.027+0000 7f58917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:32.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.027+0000 7f58917fa640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:43:32.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.027+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58941d8550 con 0x7f589414be90 2024-03-25T19:43:32.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.029+0000 7f58917fa640 10 client.?.objecter ms_handle_connect 0x7f58940b1730 2024-03-25T19:43:32.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.029+0000 7f58917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:32.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.035+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1004738428 0 0) 0x7f58941d8550 con 0x7f589414be90 2024-03-25T19:43:32.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.035+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58941d86e0 con 0x7f589414be90 2024-03-25T19:43:32.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.035+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f588c001bd0 con 0x7f589414be90 2024-03-25T19:43:32.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.035+0000 7f58917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:32.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.035+0000 7f58917fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.035+0000 7f58917fa640 10 monclient(hunting): dump: 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.035+0000 7f58917fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.035+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1570964166 0 0) 0x7f5878004eb0 con 0x7f58940b4d60 2024-03-25T19:43:32.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.035+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58941d8550 con 0x7f58940b4d60 2024-03-25T19:43:32.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.038+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396222790 0 0) 0x7f589413bc10 con 0x7f58940b1730 2024-03-25T19:43:32.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.038+0000 7f58917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:32.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.038+0000 7f58917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:32.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.038+0000 7f58917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:32.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.038+0000 7f58917fa640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:43:32.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.038+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5878004eb0 con 0x7f58940b1730 2024-03-25T19:43:32.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.049+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3490830197 0 0) 0x7f5878004eb0 con 0x7f58940b1730 2024-03-25T19:43:32.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.049+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f589413bc10 con 0x7f58940b1730 2024-03-25T19:43:32.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.049+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5888001910 con 0x7f58940b1730 2024-03-25T19:43:32.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.049+0000 7f58917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:32.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.049+0000 7f58917fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:32.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.049+0000 7f58917fa640 10 monclient(hunting): dump: 2024-03-25T19:43:32.049 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:32.049 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:32.049 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:32.050 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:32.050 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:32.050 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:32.050 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:32.050 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:32.050 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:32.050 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:32.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2563579416 0 0) 0x7f58941d86e0 con 0x7f589414be90 2024-03-25T19:43:32.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 >> v1:172.21.15.195:6789/0 conn(0x7f58940b4d60 legacy=0x7f58941d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 >> v1:172.21.15.107:6789/0 conn(0x7f58940b1730 legacy=0x7f589413a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f58917fa640 10 monclient: _finish_hunting 0 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f58917fa640 1 monclient: found mon.c 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f58917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f58917fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589413d0f0 con 0x7f589414be90 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f58917fa640 10 monclient: _finish_auth 0 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f58917fa640 10 monclient: _check_auth_tickets 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f58917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f589af32640 5 monclient: authenticate success, global_id 4574 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f589af32640 10 monclient: _renew_subs 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f589af32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f589af32640 1 -- 172.21.15.107:0/352284511 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589413d340 con 0x7f589414be90 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f589af32640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f589af32640 10 monclient: _renew_subs 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f589af32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f589af32640 1 -- 172.21.15.107:0/352284511 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58941335f0 con 0x7f589414be90 2024-03-25T19:43:32.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.050+0000 7f589af32640 1 librados: init done 2024-03-25T19:43:32.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.051+0000 7f589af32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:32.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.051+0000 7f589af32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:32.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.051+0000 7f589af32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:32.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.051+0000 7f589af32640 1 -- 172.21.15.107:0/352284511 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58941d7aa0 con 0x7f589414be90 2024-03-25T19:43:32.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.364+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f588c002080 con 0x7f589414be90 2024-03-25T19:43:32.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.364+0000 7f58917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:32.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.364+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f588c004d00 con 0x7f589414be90 2024-03-25T19:43:32.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.364+0000 7f58917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:32.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.364+0000 7f58917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:32.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.364+0000 7f58917fa640 10 monclient: dump: 2024-03-25T19:43:32.364 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:32.364 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:32.364 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:32.364 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:32.364 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:32.365 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:32.365 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:32.365 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:32.365 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:32.365 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:32.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.364+0000 7f5892ffd640 4 set_mon_vals no callback set 2024-03-25T19:43:32.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.365+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f588c01ad50 con 0x7f589414be90 2024-03-25T19:43:32.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.365+0000 7f5898cce640 10 monclient: get_auth_request con 0x7f587806e290 auth_method 0 2024-03-25T19:43:32.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.366+0000 7f58917fa640 10 client.4574.objecter ms_handle_connect 0x7f587806e290 2024-03-25T19:43:32.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.370+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f588c0548f0 con 0x7f589414be90 2024-03-25T19:43:32.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.370+0000 7f58917fa640 10 client.4574.objecter ms_dispatch 0x7f5894000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:32.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.370+0000 7f58917fa640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:32.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.370+0000 7f58917fa640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:32.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.370+0000 7f58917fa640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-25T19:43:32.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.370+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f588c050230 con 0x7f589414be90 2024-03-25T19:43:32.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.370+0000 7f58917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:32.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.370+0000 7f58917fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:32.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.583+0000 7f589af32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:32.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.583+0000 7f589af32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:32.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.583+0000 7f589af32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:32.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.583+0000 7f589af32640 1 -- 172.21.15.107:0/352284511 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58940627b0 con 0x7f589414be90 2024-03-25T19:43:32.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.610+0000 7f58917fa640 1 -- 172.21.15.107:0/352284511 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f588c052330 con 0x7f589414be90 2024-03-25T19:43:32.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.610+0000 7f58917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:32.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.610+0000 7f58917fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:32.610 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:43:32.610 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:32.613 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 10 librados: watch_flush enter 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 10 librados: watch_flush exit 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 10 client.4574.objecter successfully canceled tick 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 1 -- 172.21.15.107:0/352284511 >> v1:172.21.15.195:6800/2984408421 conn(0x7f587806e290 legacy=0x7f5878070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 10 monclient: shutdown 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 1 -- 172.21.15.107:0/352284511 >> v1:172.21.15.107:6790/0 conn(0x7f589414be90 legacy=0x7f5894132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 1 -- 172.21.15.107:0/352284511 shutdown_connections 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 1 -- 172.21.15.107:0/352284511 >> 172.21.15.107:0/352284511 conn(0x7f58940617a0 msgr2=0x7f58940b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 1 -- 172.21.15.107:0/352284511 shutdown_connections 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 1 -- 172.21.15.107:0/352284511 wait complete. 2024-03-25T19:43:32.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:32.612+0000 7f589af32640 1 librados: shutdown 2024-03-25T19:43:32.623 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:39.428 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:39.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.543+0000 7f587cfe6640 10 monclient: get_monmap_and_config 2024-03-25T19:43:39.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.543+0000 7f587cfe6640 10 monclient: build_initial_monmap 2024-03-25T19:43:39.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.543+0000 7f587cfe6640 10 monclient: monmap: 2024-03-25T19:43:39.543 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:39.543 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:39.543 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:43:39.544434+0000 2024-03-25T19:43:39.543 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:43:39.544434+0000 2024-03-25T19:43:39.543 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:39.543 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:39.543 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:39.543 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:43:39.544 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:43:39.544 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:39.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.544+0000 7f587cfe6640 1 Processor -- start 2024-03-25T19:43:39.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.544+0000 7f587cfe6640 1 -- start start 2024-03-25T19:43:39.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.544+0000 7f587cfe6640 10 monclient: init 2024-03-25T19:43:39.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.545+0000 7f587cfe6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:39.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.545+0000 7f587cfe6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:39.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.545+0000 7f587cfe6640 10 monclient(hunting): picked mon.noname-c con 0x7f58780a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:39.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.545+0000 7f587cfe6640 10 monclient(hunting): picked mon.noname-a con 0x7f587813f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:39.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.545+0000 7f587cfe6640 10 monclient(hunting): picked mon.noname-b con 0x7f58781430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:39.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.545+0000 7f587cfe6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5878142520 con 0x7f587813f710 2024-03-25T19:43:39.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.545+0000 7f587cfe6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58780b28a0 con 0x7f58780a98f0 2024-03-25T19:43:39.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.545+0000 7f587cfe6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58780b2a30 con 0x7f58781430c0 2024-03-25T19:43:39.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.545+0000 7f587cfe6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:39.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.545+0000 7f587cfe6640 10 monclient(hunting): authenticate will time out at 5011.886247s 2024-03-25T19:43:39.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.553+0000 7f587659c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f58780a98f0 0x7f58780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:46938/0 (socket says 172.21.15.107:46938) 2024-03-25T19:43:39.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.553+0000 7f587659c640 1 -- 172.21.15.107:0/3055356563 learned_addr learned my addr 172.21.15.107:0/3055356563 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:39.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.562+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941407356 0 0) 0x7f58780b28a0 con 0x7f58780a98f0 2024-03-25T19:43:39.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.562+0000 7f587559a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:39.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.562+0000 7f587559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:39.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.562+0000 7f587559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:39.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.562+0000 7f587559a640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:43:39.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.562+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5860003520 con 0x7f58780a98f0 2024-03-25T19:43:39.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.575+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718308993 0 0) 0x7f5878142520 con 0x7f587813f710 2024-03-25T19:43:39.575 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.575+0000 7f587559a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:39.575 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.575+0000 7f587559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:39.575 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.575+0000 7f587559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:39.575 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.575+0000 7f587559a640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:43:39.575 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.575+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58780b28a0 con 0x7f587813f710 2024-03-25T19:43:39.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.580+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051469109 0 0) 0x7f58780b2a30 con 0x7f58781430c0 2024-03-25T19:43:39.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.580+0000 7f587559a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:39.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.580+0000 7f587559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:39.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.580+0000 7f587559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:39.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.580+0000 7f587559a640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:43:39.580 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.580+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5878142520 con 0x7f58781430c0 2024-03-25T19:43:39.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2090475909 0 0) 0x7f5860003520 con 0x7f58780a98f0 2024-03-25T19:43:39.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 >> v1:172.21.15.107:6790/0 conn(0x7f58781430c0 legacy=0x7f587814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 >> v1:172.21.15.107:6789/0 conn(0x7f587813f710 legacy=0x7f58781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 10 monclient: _finish_hunting 0 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 1 monclient: found mon.noname-c 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58780b44c0 con 0x7f58780a98f0 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 10 monclient: _finish_auth 0 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 10 monclient: _check_auth_tickets 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5868001690 con 0x7f58780a98f0 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587cfe6640 5 monclient: authenticate success, global_id 4564 2024-03-25T19:43:39.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587cfe6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587559a640 10 monclient: dump: 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:39.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.581+0000 7f587cfe6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:39.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.602+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58680017f0 con 0x7f58780a98f0 2024-03-25T19:43:39.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.602+0000 7f587559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:39.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.602+0000 7f587559a640 1 -- 172.21.15.107:0/3055356563 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5868001c00 con 0x7f58780a98f0 2024-03-25T19:43:39.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.602+0000 7f587559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:39.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.602+0000 7f587559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:39.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.602+0000 7f587559a640 10 monclient: dump: 2024-03-25T19:43:39.602 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:39.602 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:39.602 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.602 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.602 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:39.603 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:39.603 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:39.603 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:39.603 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:39.603 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:39.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.602+0000 7f587cfe6640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:39.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.602+0000 7f587cfe6640 4 set_mon_vals no callback set 2024-03-25T19:43:39.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.603+0000 7f587cfe6640 10 monclient: shutdown 2024-03-25T19:43:39.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.603+0000 7f587cfe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:39.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.603+0000 7f587cfe6640 1 -- 172.21.15.107:0/3055356563 >> v1:172.21.15.195:6789/0 conn(0x7f58780a98f0 legacy=0x7f58780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.603+0000 7f587cfe6640 1 -- 172.21.15.107:0/3055356563 shutdown_connections 2024-03-25T19:43:39.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.603+0000 7f587cfe6640 1 -- 172.21.15.107:0/3055356563 >> 172.21.15.107:0/3055356563 conn(0x7f58780b7ea0 msgr2=0x7f58780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:39.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.603+0000 7f587cfe6640 1 -- 172.21.15.107:0/3055356563 shutdown_connections 2024-03-25T19:43:39.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.603+0000 7f587cfe6640 1 -- 172.21.15.107:0/3055356563 wait complete. 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.604+0000 7f587cfe6640 10 monclient: build_initial_monmap 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.604+0000 7f587cfe6640 10 monclient: monmap: 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.604+0000 7f587cfe6640 1 librados: starting msgr at 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.604+0000 7f587cfe6640 1 librados: starting objecter 2024-03-25T19:43:39.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 1 Processor -- start 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 1 -- start start 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 1 librados: setting wanted keys 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 1 librados: calling monclient init 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 10 monclient: init 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 10 monclient(hunting): picked mon.noname-a con 0x7f58780a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 10 monclient(hunting): picked mon.noname-b con 0x7f587813f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 10 monclient(hunting): picked mon.noname-c con 0x7f58781430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5878135590 con 0x7f58780a98f0 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5878136e50 con 0x7f587813f710 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5878135070 con 0x7f58781430c0 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:39.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.605+0000 7f587cfe6640 10 monclient(hunting): authenticate will time out at 5011.946402s 2024-03-25T19:43:39.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.611+0000 7f5876d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f58781430c0 0x7f58781e3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:58120/0 (socket says 172.21.15.107:58120) 2024-03-25T19:43:39.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.612+0000 7f5876d9d640 1 -- 172.21.15.107:0/2494263099 learned_addr learned my addr 172.21.15.107:0/2494263099 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:39.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.613+0000 7f5866ffd640 10 client.?.objecter ms_handle_connect 0x7f58781430c0 2024-03-25T19:43:39.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.613+0000 7f5866ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:39.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.618+0000 7f5866ffd640 10 client.?.objecter ms_handle_connect 0x7f58780a98f0 2024-03-25T19:43:39.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.618+0000 7f5866ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:39.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.618+0000 7f5866ffd640 10 client.?.objecter ms_handle_connect 0x7f587813f710 2024-03-25T19:43:39.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.618+0000 7f5866ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:39.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.623+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449533208 0 0) 0x7f5878135070 con 0x7f58781430c0 2024-03-25T19:43:39.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.623+0000 7f5866ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:39.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.623+0000 7f5866ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:39.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.623+0000 7f5866ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:39.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.623+0000 7f5866ffd640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:43:39.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.623+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5858004eb0 con 0x7f58781430c0 2024-03-25T19:43:39.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.634+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615342280 0 0) 0x7f5878136e50 con 0x7f587813f710 2024-03-25T19:43:39.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.634+0000 7f5866ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:39.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.634+0000 7f5866ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:39.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.634+0000 7f5866ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:39.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.634+0000 7f5866ffd640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:43:39.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.634+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5878135070 con 0x7f587813f710 2024-03-25T19:43:39.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.635+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759566191 0 0) 0x7f5878135590 con 0x7f58780a98f0 2024-03-25T19:43:39.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.635+0000 7f5866ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:39.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.635+0000 7f5866ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:39.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.635+0000 7f5866ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:39.635 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.635+0000 7f5866ffd640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:43:39.635 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.635+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5878136e50 con 0x7f58780a98f0 2024-03-25T19:43:39.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.642+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1419225042 0 0) 0x7f5858004eb0 con 0x7f58781430c0 2024-03-25T19:43:39.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.642+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5878135590 con 0x7f58781430c0 2024-03-25T19:43:39.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.642+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5870001c70 con 0x7f58781430c0 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.642+0000 7f5866ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.642+0000 7f5866ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.642+0000 7f5866ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:39.642 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:39.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.642+0000 7f5866ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:39.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.653+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3472606754 0 0) 0x7f5878136e50 con 0x7f58780a98f0 2024-03-25T19:43:39.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.653+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5858004eb0 con 0x7f58780a98f0 2024-03-25T19:43:39.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.653+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5868001b00 con 0x7f58780a98f0 2024-03-25T19:43:39.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.653+0000 7f5866ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:39.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.653+0000 7f5866ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:39.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.653+0000 7f5866ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:39.653 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:39.654 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:39.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.660+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3921662744 0 0) 0x7f5878135070 con 0x7f587813f710 2024-03-25T19:43:39.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.660+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5878136e50 con 0x7f587813f710 2024-03-25T19:43:39.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.660+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f586c001930 con 0x7f587813f710 2024-03-25T19:43:39.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.660+0000 7f5866ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:39.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.660+0000 7f5866ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:39.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.660+0000 7f5866ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:39.660 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:39.660 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:39.660 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.660 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.660 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:39.661 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:39.661 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:39.661 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:39.661 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:39.661 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:39.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2185225095 0 0) 0x7f5878135590 con 0x7f58781430c0 2024-03-25T19:43:39.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 >> v1:172.21.15.195:6789/0 conn(0x7f587813f710 legacy=0x7f58781d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 >> v1:172.21.15.107:6789/0 conn(0x7f58780a98f0 legacy=0x7f5878130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f5866ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f5866ffd640 1 monclient: found mon.c 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f5866ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f5866ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5878135d80 con 0x7f58781430c0 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f5866ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f5866ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f5866ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f587cfe6640 5 monclient: authenticate success, global_id 4580 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f587cfe6640 10 monclient: _renew_subs 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f587cfe6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f587cfe6640 1 -- 172.21.15.107:0/2494263099 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5878131660 con 0x7f58781430c0 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f587cfe6640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f587cfe6640 10 monclient: _renew_subs 2024-03-25T19:43:39.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f587cfe6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f587cfe6640 1 -- 172.21.15.107:0/2494263099 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58780b78a0 con 0x7f58781430c0 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.666+0000 7f587cfe6640 1 librados: init done 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.667+0000 7f587cfe6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.667+0000 7f587cfe6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.667+0000 7f587cfe6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:39.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.667+0000 7f587cfe6640 1 -- 172.21.15.107:0/2494263099 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5878135f70 con 0x7f58781430c0 2024-03-25T19:43:39.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.973+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5870003990 con 0x7f58781430c0 2024-03-25T19:43:39.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.973+0000 7f5866ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:39.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.973+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5870003c00 con 0x7f58781430c0 2024-03-25T19:43:39.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.973+0000 7f5866ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:39.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.973+0000 7f5866ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:39.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.973+0000 7f5866ffd640 10 monclient: dump: 2024-03-25T19:43:39.974 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:39.974 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:39.974 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.974 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:39.974 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:39.974 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:39.974 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:39.974 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:39.974 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:39.974 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:39.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.973+0000 7f5874d99640 4 set_mon_vals no callback set 2024-03-25T19:43:39.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.974+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f5870003e20 con 0x7f58781430c0 2024-03-25T19:43:39.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.975+0000 7f587659c640 10 monclient: get_auth_request con 0x7f585806e160 auth_method 0 2024-03-25T19:43:39.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.975+0000 7f5866ffd640 10 client.4580.objecter ms_handle_connect 0x7f585806e160 2024-03-25T19:43:39.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.980+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f5870054920 con 0x7f58781430c0 2024-03-25T19:43:39.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.980+0000 7f5866ffd640 10 client.4580.objecter ms_dispatch 0x7f5878000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:39.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.980+0000 7f5866ffd640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:39.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.980+0000 7f5866ffd640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:39.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.980+0000 7f5866ffd640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-25T19:43:39.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.980+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5870050430 con 0x7f58781430c0 2024-03-25T19:43:39.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.980+0000 7f5866ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:39.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:39.980+0000 7f5866ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:40.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.226+0000 7f587cfe6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:40.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.226+0000 7f587cfe6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:40.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.226+0000 7f587cfe6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:40.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.226+0000 7f587cfe6640 1 -- 172.21.15.107:0/2494263099 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5878135200 con 0x7f58781430c0 2024-03-25T19:43:40.250 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.250+0000 7f5866ffd640 1 -- 172.21.15.107:0/2494263099 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f5870052360 con 0x7f58781430c0 2024-03-25T19:43:40.250 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.250+0000 7f5866ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:40.250 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.250+0000 7f5866ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:40.251 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:43:40.251 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.252+0000 7f587cfe6640 10 librados: watch_flush enter 2024-03-25T19:43:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.252+0000 7f587cfe6640 10 librados: watch_flush exit 2024-03-25T19:43:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.252+0000 7f587cfe6640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-25T19:43:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.252+0000 7f587cfe6640 10 client.4580.objecter successfully canceled tick 2024-03-25T19:43:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.252+0000 7f587cfe6640 1 -- 172.21.15.107:0/2494263099 >> v1:172.21.15.195:6800/2984408421 conn(0x7f585806e160 legacy=0x7f5858070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.253+0000 7f587cfe6640 10 monclient: shutdown 2024-03-25T19:43:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.253+0000 7f587cfe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.253+0000 7f587cfe6640 1 -- 172.21.15.107:0/2494263099 >> v1:172.21.15.107:6790/0 conn(0x7f58781430c0 legacy=0x7f58781e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.253+0000 7f587cfe6640 1 -- 172.21.15.107:0/2494263099 shutdown_connections 2024-03-25T19:43:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.253+0000 7f587cfe6640 1 -- 172.21.15.107:0/2494263099 >> 172.21.15.107:0/2494263099 conn(0x7f58780b7ea0 msgr2=0x7f5878074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:40.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.253+0000 7f587cfe6640 1 -- 172.21.15.107:0/2494263099 shutdown_connections 2024-03-25T19:43:40.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.253+0000 7f587cfe6640 1 -- 172.21.15.107:0/2494263099 wait complete. 2024-03-25T19:43:40.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:40.253+0000 7f587cfe6640 1 librados: shutdown 2024-03-25T19:43:40.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:45.719 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:08:16.035 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:45.724 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:08:16.041 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:45.757 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:08:16.073 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:45.777 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:08:16.064 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:45.783 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:08:16.099 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:45.785 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:08:16.072 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:45.793 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:08:16.109 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:45.820 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:08:16.136 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:45.825 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:08:16.112 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:47.070 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:47.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.182+0000 7ff87c84a640 10 monclient: get_monmap_and_config 2024-03-25T19:43:47.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.182+0000 7ff87c84a640 10 monclient: build_initial_monmap 2024-03-25T19:43:47.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.182+0000 7ff87c84a640 10 monclient: monmap: 2024-03-25T19:43:47.182 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:47.182 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:47.183 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:43:47.183872+0000 2024-03-25T19:43:47.183 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:43:47.183872+0000 2024-03-25T19:43:47.183 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:47.183 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:47.183 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:47.183 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:43:47.183 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:43:47.183 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:47.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.183+0000 7ff87c84a640 1 Processor -- start 2024-03-25T19:43:47.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.183+0000 7ff87c84a640 1 -- start start 2024-03-25T19:43:47.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.183+0000 7ff87c84a640 10 monclient: init 2024-03-25T19:43:47.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.184+0000 7ff87c84a640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:47.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.184+0000 7ff87c84a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:47.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.184+0000 7ff87c84a640 10 monclient(hunting): picked mon.noname-c con 0x7ff8780a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:47.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.184+0000 7ff87c84a640 10 monclient(hunting): picked mon.noname-a con 0x7ff87813f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:47.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.184+0000 7ff87c84a640 10 monclient(hunting): picked mon.noname-b con 0x7ff8781430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:47.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.184+0000 7ff87c84a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff878142520 con 0x7ff87813f710 2024-03-25T19:43:47.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.184+0000 7ff87c84a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8780b28a0 con 0x7ff8780a98f0 2024-03-25T19:43:47.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.184+0000 7ff87c84a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8780b2a30 con 0x7ff8781430c0 2024-03-25T19:43:47.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.184+0000 7ff87c84a640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:47.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.184+0000 7ff87c84a640 10 monclient(hunting): authenticate will time out at 5019.525552s 2024-03-25T19:43:47.194 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.194+0000 7ff876d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff8781430c0 0x7ff87814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44310/0 (socket says 172.21.15.107:44310) 2024-03-25T19:43:47.194 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.195+0000 7ff876d9d640 1 -- 172.21.15.107:0/3392245907 learned_addr learned my addr 172.21.15.107:0/3392245907 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:47.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.207+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165537156 0 0) 0x7ff8780b2a30 con 0x7ff8781430c0 2024-03-25T19:43:47.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.207+0000 7ff87559a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:47.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.207+0000 7ff87559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:47.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.207+0000 7ff87559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:47.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.207+0000 7ff87559a640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:43:47.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.207+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff860003520 con 0x7ff8781430c0 2024-03-25T19:43:47.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.216+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737538533 0 0) 0x7ff8780b28a0 con 0x7ff8780a98f0 2024-03-25T19:43:47.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.216+0000 7ff87559a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:47.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.216+0000 7ff87559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:47.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.216+0000 7ff87559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:47.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.216+0000 7ff87559a640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:43:47.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.216+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8780b2a30 con 0x7ff8780a98f0 2024-03-25T19:43:47.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.220+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218555784 0 0) 0x7ff878142520 con 0x7ff87813f710 2024-03-25T19:43:47.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.220+0000 7ff87559a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:47.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.220+0000 7ff87559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:47.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.220+0000 7ff87559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:47.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.220+0000 7ff87559a640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:43:47.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.220+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8780b28a0 con 0x7ff87813f710 2024-03-25T19:43:47.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1820524509 0 0) 0x7ff8780b2a30 con 0x7ff8780a98f0 2024-03-25T19:43:47.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 >> v1:172.21.15.107:6790/0 conn(0x7ff8781430c0 legacy=0x7ff87814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 >> v1:172.21.15.107:6789/0 conn(0x7ff87813f710 legacy=0x7ff8781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 10 monclient: _finish_hunting 0 2024-03-25T19:43:47.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 1 monclient: found mon.noname-c 2024-03-25T19:43:47.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:47.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:43:47.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8780b44c0 con 0x7ff8780a98f0 2024-03-25T19:43:47.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 10 monclient: _finish_auth 0 2024-03-25T19:43:47.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 10 monclient: _check_auth_tickets 2024-03-25T19:43:47.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff868001670 con 0x7ff8780a98f0 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87559a640 10 monclient: dump: 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87c84a640 5 monclient: authenticate success, global_id 4570 2024-03-25T19:43:47.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.231+0000 7ff87c84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.251+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8680017d0 con 0x7ff8780a98f0 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.251+0000 7ff87559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.251+0000 7ff87559a640 1 -- 172.21.15.107:0/3392245907 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff868001be0 con 0x7ff8780a98f0 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.251+0000 7ff87559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.251+0000 7ff87559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.251+0000 7ff87559a640 10 monclient: dump: 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:47.251 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.251+0000 7ff87c84a640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:47.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.251+0000 7ff87c84a640 4 set_mon_vals no callback set 2024-03-25T19:43:47.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.252+0000 7ff87c84a640 10 monclient: shutdown 2024-03-25T19:43:47.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.252+0000 7ff87c84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:47.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.252+0000 7ff87c84a640 1 -- 172.21.15.107:0/3392245907 >> v1:172.21.15.195:6789/0 conn(0x7ff8780a98f0 legacy=0x7ff8780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.252+0000 7ff87c84a640 1 -- 172.21.15.107:0/3392245907 shutdown_connections 2024-03-25T19:43:47.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.252+0000 7ff87c84a640 1 -- 172.21.15.107:0/3392245907 >> 172.21.15.107:0/3392245907 conn(0x7ff8780b7ea0 msgr2=0x7ff8780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.252+0000 7ff87c84a640 1 -- 172.21.15.107:0/3392245907 shutdown_connections 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.252+0000 7ff87c84a640 1 -- 172.21.15.107:0/3392245907 wait complete. 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.253+0000 7ff87c84a640 10 monclient: build_initial_monmap 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.253+0000 7ff87c84a640 10 monclient: monmap: 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:43:47.255 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.253+0000 7ff87c84a640 1 librados: starting msgr at 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.253+0000 7ff87c84a640 1 librados: starting objecter 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.253+0000 7ff87c84a640 1 Processor -- start 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.253+0000 7ff87c84a640 1 -- start start 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.253+0000 7ff87c84a640 1 librados: setting wanted keys 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.253+0000 7ff87c84a640 1 librados: calling monclient init 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.253+0000 7ff87c84a640 10 monclient: init 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.254+0000 7ff87c84a640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.254+0000 7ff87c84a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.254+0000 7ff87c84a640 10 monclient(hunting): picked mon.noname-b con 0x7ff8780a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.254+0000 7ff87c84a640 10 monclient(hunting): picked mon.noname-c con 0x7ff87813f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.254+0000 7ff87c84a640 10 monclient(hunting): picked mon.noname-a con 0x7ff8781430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.254+0000 7ff87c84a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff878137f50 con 0x7ff8781430c0 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.254+0000 7ff87c84a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff878139830 con 0x7ff8780a98f0 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.254+0000 7ff87c84a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8780aba50 con 0x7ff87813f710 2024-03-25T19:43:47.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.254+0000 7ff87c84a640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:47.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.254+0000 7ff87c84a640 10 monclient(hunting): authenticate will time out at 5019.595473s 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.262+0000 7ff876d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff8781430c0 0x7ff878136240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34970/0 (socket says 172.21.15.107:34970) 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.262+0000 7ff876d9d640 1 -- 172.21.15.107:0/551262820 learned_addr learned my addr 172.21.15.107:0/551262820 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:47.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.262+0000 7ff866ffd640 10 client.?.objecter ms_handle_connect 0x7ff8781430c0 2024-03-25T19:43:47.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.262+0000 7ff866ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:47.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.274+0000 7ff866ffd640 10 client.?.objecter ms_handle_connect 0x7ff87813f710 2024-03-25T19:43:47.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.274+0000 7ff866ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:47.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.277+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871165576 0 0) 0x7ff878137f50 con 0x7ff8781430c0 2024-03-25T19:43:47.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.277+0000 7ff866ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:47.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.277+0000 7ff866ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:47.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.277+0000 7ff866ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:47.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.277+0000 7ff866ffd640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:43:47.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.277+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff858004eb0 con 0x7ff8781430c0 2024-03-25T19:43:47.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.277+0000 7ff866ffd640 10 client.?.objecter ms_handle_connect 0x7ff8780a98f0 2024-03-25T19:43:47.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.277+0000 7ff866ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:47.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.283+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679676068 0 0) 0x7ff878139830 con 0x7ff8780a98f0 2024-03-25T19:43:47.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.283+0000 7ff866ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:47.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.283+0000 7ff866ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:47.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.283+0000 7ff866ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:47.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.283+0000 7ff866ffd640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:43:47.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.283+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff878137f50 con 0x7ff8780a98f0 2024-03-25T19:43:47.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.287+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408912397 0 0) 0x7ff8780aba50 con 0x7ff87813f710 2024-03-25T19:43:47.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.287+0000 7ff866ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:47.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.287+0000 7ff866ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:47.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.287+0000 7ff866ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:47.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.287+0000 7ff866ffd640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:43:47.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.288+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff878139830 con 0x7ff87813f710 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.308+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2827754266 0 0) 0x7ff858004eb0 con 0x7ff8781430c0 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.308+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8780aba50 con 0x7ff8781430c0 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.308+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff870001b30 con 0x7ff8781430c0 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.308+0000 7ff866ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.308+0000 7ff866ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.308+0000 7ff866ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:47.308 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:47.309 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:47.309 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:47.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.308+0000 7ff866ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:47.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.309+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070551868 0 0) 0x7ff878137f50 con 0x7ff8780a98f0 2024-03-25T19:43:47.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.309+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff858004eb0 con 0x7ff8780a98f0 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.309+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff868001ae0 con 0x7ff8780a98f0 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.309+0000 7ff866ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.309+0000 7ff866ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.309+0000 7ff866ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:47.310 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:47.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.314+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2183882322 0 0) 0x7ff878139830 con 0x7ff87813f710 2024-03-25T19:43:47.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.314+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff878137f50 con 0x7ff87813f710 2024-03-25T19:43:47.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.314+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff86c001830 con 0x7ff87813f710 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.314+0000 7ff866ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.314+0000 7ff866ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.314+0000 7ff866ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:47.314 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:47.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3895989109 0 0) 0x7ff878137f50 con 0x7ff87813f710 2024-03-25T19:43:47.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 >> v1:172.21.15.195:6789/0 conn(0x7ff8780a98f0 legacy=0x7ff878134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 >> v1:172.21.15.107:6789/0 conn(0x7ff8781430c0 legacy=0x7ff878136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff866ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:47.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff866ffd640 1 monclient: found mon.c 2024-03-25T19:43:47.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff866ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:47.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff866ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:47.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8780abd30 con 0x7ff87813f710 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff866ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff866ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff866ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff87c84a640 5 monclient: authenticate success, global_id 4586 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff87c84a640 10 monclient: _renew_subs 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff87c84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff87c84a640 1 -- 172.21.15.107:0/551262820 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff878137a00 con 0x7ff87813f710 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff87c84a640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff87c84a640 10 monclient: _renew_subs 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff87c84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff87c84a640 1 -- 172.21.15.107:0/551262820 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff878134970 con 0x7ff87813f710 2024-03-25T19:43:47.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.323+0000 7ff87c84a640 1 librados: init done 2024-03-25T19:43:47.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.324+0000 7ff87c84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:47.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.324+0000 7ff87c84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:47.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.324+0000 7ff87c84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:47.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.324+0000 7ff87c84a640 1 -- 172.21.15.107:0/551262820 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8780627b0 con 0x7ff87813f710 2024-03-25T19:43:47.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.632+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff86c001d40 con 0x7ff87813f710 2024-03-25T19:43:47.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.632+0000 7ff866ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:47.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.632+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff86c0037f0 con 0x7ff87813f710 2024-03-25T19:43:47.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.632+0000 7ff866ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:47.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.632+0000 7ff866ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:47.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.632+0000 7ff866ffd640 10 monclient: dump: 2024-03-25T19:43:47.632 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:47.632 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:47.632 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.632 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:47.632 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:47.633 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:47.633 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:47.633 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:47.633 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:47.633 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:47.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.632+0000 7ff874d99640 4 set_mon_vals no callback set 2024-03-25T19:43:47.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.633+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7ff86c019880 con 0x7ff87813f710 2024-03-25T19:43:47.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.633+0000 7ff87659c640 10 monclient: get_auth_request con 0x7ff85806e160 auth_method 0 2024-03-25T19:43:47.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.634+0000 7ff866ffd640 10 client.4586.objecter ms_handle_connect 0x7ff85806e160 2024-03-25T19:43:47.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.638+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7ff86c053560 con 0x7ff87813f710 2024-03-25T19:43:47.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.638+0000 7ff866ffd640 10 client.4586.objecter ms_dispatch 0x7ff878000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:47.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.638+0000 7ff866ffd640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:47.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.638+0000 7ff866ffd640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:47.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.638+0000 7ff866ffd640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-25T19:43:47.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.638+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff86c04eea0 con 0x7ff87813f710 2024-03-25T19:43:47.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.638+0000 7ff866ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:47.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.638+0000 7ff866ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:47.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.882+0000 7ff87c84a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:47.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.882+0000 7ff87c84a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:47.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.882+0000 7ff87c84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:47.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.882+0000 7ff87c84a640 1 -- 172.21.15.107:0/551262820 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff878135850 con 0x7ff87813f710 2024-03-25T19:43:47.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.921+0000 7ff866ffd640 1 -- 172.21.15.107:0/551262820 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7ff878135850 con 0x7ff87813f710 2024-03-25T19:43:47.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.921+0000 7ff866ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:47.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.921+0000 7ff866ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:47.921 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:43:47.921 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:47.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 10 librados: watch_flush enter 2024-03-25T19:43:47.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 10 librados: watch_flush exit 2024-03-25T19:43:47.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-25T19:43:47.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 10 client.4586.objecter successfully canceled tick 2024-03-25T19:43:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 1 -- 172.21.15.107:0/551262820 >> v1:172.21.15.195:6800/2984408421 conn(0x7ff85806e160 legacy=0x7ff858070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 10 monclient: shutdown 2024-03-25T19:43:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 1 -- 172.21.15.107:0/551262820 >> v1:172.21.15.107:6790/0 conn(0x7ff87813f710 legacy=0x7ff878135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 1 -- 172.21.15.107:0/551262820 shutdown_connections 2024-03-25T19:43:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 1 -- 172.21.15.107:0/551262820 >> 172.21.15.107:0/551262820 conn(0x7ff8780b7ea0 msgr2=0x7ff878074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 1 -- 172.21.15.107:0/551262820 shutdown_connections 2024-03-25T19:43:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 1 -- 172.21.15.107:0/551262820 wait complete. 2024-03-25T19:43:47.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:47.923+0000 7ff87c84a640 1 librados: shutdown 2024-03-25T19:43:47.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:54.739 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:54.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.856+0000 7faa59ed6640 10 monclient: get_monmap_and_config 2024-03-25T19:43:54.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.856+0000 7faa59ed6640 10 monclient: build_initial_monmap 2024-03-25T19:43:54.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.856+0000 7faa59ed6640 10 monclient: monmap: 2024-03-25T19:43:54.856 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:54.856 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:54.856 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:43:54.857621+0000 2024-03-25T19:43:54.856 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:43:54.857621+0000 2024-03-25T19:43:54.856 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:54.856 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:54.856 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:54.857 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:43:54.857 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:43:54.857 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.857+0000 7faa59ed6640 1 Processor -- start 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.857+0000 7faa59ed6640 1 -- start start 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.857+0000 7faa59ed6640 10 monclient: init 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.858+0000 7faa59ed6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.858+0000 7faa59ed6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.858+0000 7faa59ed6640 10 monclient(hunting): picked mon.noname-c con 0x7faa54061830 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.858+0000 7faa59ed6640 10 monclient(hunting): picked mon.noname-b con 0x7faa540ae900 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:54.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.858+0000 7faa59ed6640 10 monclient(hunting): picked mon.noname-a con 0x7faa541440e0 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.858+0000 7faa59ed6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5406f4f0 con 0x7faa541440e0 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.858+0000 7faa59ed6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa54070c40 con 0x7faa54061830 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.858+0000 7faa59ed6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa54070eb0 con 0x7faa540ae900 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.858+0000 7faa59ed6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:54.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.858+0000 7faa59ed6640 10 monclient(hunting): authenticate will time out at 5027.199296s 2024-03-25T19:43:54.870 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.870+0000 7faa537fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7faa54061830 0x7faa54146710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:43382/0 (socket says 172.21.15.107:43382) 2024-03-25T19:43:54.870 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.870+0000 7faa537fe640 1 -- 172.21.15.107:0/2864313816 learned_addr learned my addr 172.21.15.107:0/2864313816 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:54.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.882+0000 7faa527fc640 1 -- 172.21.15.107:0/2864313816 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599056793 0 0) 0x7faa54070c40 con 0x7faa54061830 2024-03-25T19:43:54.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.882+0000 7faa527fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:54.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.882+0000 7faa527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:54.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.882+0000 7faa527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:54.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.882+0000 7faa527fc640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:43:54.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.882+0000 7faa527fc640 1 -- 172.21.15.107:0/2864313816 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa3c003520 con 0x7faa54061830 2024-03-25T19:43:54.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.891+0000 7faa527fc640 1 -- 172.21.15.107:0/2864313816 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027409840 0 0) 0x7faa54070eb0 con 0x7faa540ae900 2024-03-25T19:43:54.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.891+0000 7faa527fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:54.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.891+0000 7faa527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:54.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.891+0000 7faa527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:54.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.891+0000 7faa527fc640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:43:54.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.891+0000 7faa527fc640 1 -- 172.21.15.107:0/2864313816 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa54070c40 con 0x7faa540ae900 2024-03-25T19:43:54.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 1 -- 172.21.15.107:0/2864313816 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3052518544 0 0) 0x7faa3c003520 con 0x7faa54061830 2024-03-25T19:43:54.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 1 -- 172.21.15.107:0/2864313816 >> v1:172.21.15.107:6790/0 conn(0x7faa540ae900 legacy=0x7faa541439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 1 -- 172.21.15.107:0/2864313816 >> v1:172.21.15.107:6789/0 conn(0x7faa541440e0 legacy=0x7faa54151f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:54.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 1 monclient: found mon.noname-c 2024-03-25T19:43:54.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 1 -- 172.21.15.107:0/2864313816 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5406e730 con 0x7faa54061830 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 1 -- 172.21.15.107:0/2864313816 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faa44001690 con 0x7faa54061830 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa59ed6640 5 monclient: authenticate success, global_id 4576 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa59ed6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa527fc640 10 monclient: dump: 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:54.901 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:54.902 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.902 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:54.902 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:54.902 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:54.902 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:54.902 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:54.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.900+0000 7faa59ed6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:54.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.923+0000 7faa527fc640 1 -- 172.21.15.107:0/2864313816 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa440017f0 con 0x7faa54061830 2024-03-25T19:43:54.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.923+0000 7faa527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.923+0000 7faa527fc640 1 -- 172.21.15.107:0/2864313816 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faa44001c00 con 0x7faa54061830 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.923+0000 7faa527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.923+0000 7faa527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.923+0000 7faa527fc640 10 monclient: dump: 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:54.923 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:54.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.923+0000 7faa59ed6640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:54.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.923+0000 7faa59ed6640 4 set_mon_vals no callback set 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.924+0000 7faa59ed6640 10 monclient: shutdown 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.924+0000 7faa59ed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.924+0000 7faa59ed6640 1 -- 172.21.15.107:0/2864313816 >> v1:172.21.15.195:6789/0 conn(0x7faa54061830 legacy=0x7faa54146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.924+0000 7faa59ed6640 1 -- 172.21.15.107:0/2864313816 shutdown_connections 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.924+0000 7faa59ed6640 1 -- 172.21.15.107:0/2864313816 >> 172.21.15.107:0/2864313816 conn(0x7faa540b0ef0 msgr2=0x7faa540b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.924+0000 7faa59ed6640 1 -- 172.21.15.107:0/2864313816 shutdown_connections 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.924+0000 7faa59ed6640 1 -- 172.21.15.107:0/2864313816 wait complete. 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.925+0000 7faa59ed6640 10 monclient: build_initial_monmap 2024-03-25T19:43:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.925+0000 7faa59ed6640 10 monclient: monmap: 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.925+0000 7faa59ed6640 1 librados: starting msgr at 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.925+0000 7faa59ed6640 1 librados: starting objecter 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.925+0000 7faa59ed6640 1 Processor -- start 2024-03-25T19:43:54.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.925+0000 7faa59ed6640 1 -- start start 2024-03-25T19:43:54.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.925+0000 7faa59ed6640 1 librados: setting wanted keys 2024-03-25T19:43:54.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.925+0000 7faa59ed6640 1 librados: calling monclient init 2024-03-25T19:43:54.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.925+0000 7faa59ed6640 10 monclient: init 2024-03-25T19:43:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.926+0000 7faa59ed6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.926+0000 7faa59ed6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.926+0000 7faa59ed6640 10 monclient(hunting): picked mon.noname-a con 0x7faa54061830 addr v1:172.21.15.107:6789/0 2024-03-25T19:43:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.926+0000 7faa59ed6640 10 monclient(hunting): picked mon.noname-c con 0x7faa541440e0 addr v1:172.21.15.107:6790/0 2024-03-25T19:43:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.926+0000 7faa59ed6640 10 monclient(hunting): picked mon.noname-b con 0x7faa540ad700 addr v1:172.21.15.195:6789/0 2024-03-25T19:43:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.926+0000 7faa59ed6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa540a9be0 con 0x7faa54061830 2024-03-25T19:43:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.926+0000 7faa59ed6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa540aafa0 con 0x7faa540ad700 2024-03-25T19:43:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.926+0000 7faa59ed6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa540a9470 con 0x7faa541440e0 2024-03-25T19:43:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.926+0000 7faa59ed6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.926+0000 7faa59ed6640 10 monclient(hunting): authenticate will time out at 5027.267605s 2024-03-25T19:43:54.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.938+0000 7faa52ffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7faa541440e0 0x7faa5413ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:32942/0 (socket says 172.21.15.107:32942) 2024-03-25T19:43:54.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.938+0000 7faa52ffd640 1 -- 172.21.15.107:0/1014224940 learned_addr learned my addr 172.21.15.107:0/1014224940 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:43:54.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.939+0000 7faa3bfff640 10 client.?.objecter ms_handle_connect 0x7faa541440e0 2024-03-25T19:43:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.939+0000 7faa3bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:54.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.947+0000 7faa3bfff640 10 client.?.objecter ms_handle_connect 0x7faa540ad700 2024-03-25T19:43:54.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.947+0000 7faa3bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:54.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.953+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130962512 0 0) 0x7faa540a9470 con 0x7faa541440e0 2024-03-25T19:43:54.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.953+0000 7faa3bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:54.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.953+0000 7faa3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:54.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.953+0000 7faa3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:54.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.953+0000 7faa3bfff640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:43:54.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.953+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa40004eb0 con 0x7faa541440e0 2024-03-25T19:43:54.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.953+0000 7faa3bfff640 10 client.?.objecter ms_handle_connect 0x7faa54061830 2024-03-25T19:43:54.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.953+0000 7faa3bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:54.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.956+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753263773 0 0) 0x7faa540aafa0 con 0x7faa540ad700 2024-03-25T19:43:54.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.956+0000 7faa3bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:54.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.956+0000 7faa3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:54.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.956+0000 7faa3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:54.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.956+0000 7faa3bfff640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:43:54.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.956+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa540a9470 con 0x7faa540ad700 2024-03-25T19:43:54.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.958+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600566752 0 0) 0x7faa540a9be0 con 0x7faa54061830 2024-03-25T19:43:54.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.958+0000 7faa3bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:54.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.958+0000 7faa3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:54.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.958+0000 7faa3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:54.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.958+0000 7faa3bfff640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:43:54.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.958+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa540aafa0 con 0x7faa54061830 2024-03-25T19:43:54.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.965+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 852657660 0 0) 0x7faa40004eb0 con 0x7faa541440e0 2024-03-25T19:43:54.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.965+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa540a9be0 con 0x7faa541440e0 2024-03-25T19:43:54.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.965+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faa48001a10 con 0x7faa541440e0 2024-03-25T19:43:54.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.965+0000 7faa3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.965+0000 7faa3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:54.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.965+0000 7faa3bfff640 10 monclient(hunting): dump: 2024-03-25T19:43:54.965 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:54.965 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:54.965 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:54.966 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:54.966 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.966 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:54.966 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:54.966 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:54.966 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:54.966 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:54.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.965+0000 7faa3bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.966+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1703665869 0 0) 0x7faa540aafa0 con 0x7faa54061830 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.966+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa40004eb0 con 0x7faa54061830 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.966+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faa44001b00 con 0x7faa54061830 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.966+0000 7faa3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.966+0000 7faa3bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.966+0000 7faa3bfff640 10 monclient(hunting): dump: 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:54.967 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:54.968 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:54.968 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:54.968 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:54.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.973+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1106335605 0 0) 0x7faa540a9470 con 0x7faa540ad700 2024-03-25T19:43:54.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.973+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa540aafa0 con 0x7faa540ad700 2024-03-25T19:43:54.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.973+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faa4c001b00 con 0x7faa540ad700 2024-03-25T19:43:54.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.973+0000 7faa3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.973+0000 7faa3bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:54.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.973+0000 7faa3bfff640 10 monclient(hunting): dump: 2024-03-25T19:43:54.973 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:54.973 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:54.973 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:54.974 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:54.974 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.974 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:54.974 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:54.974 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:54.974 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:54.974 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:54.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3397559673 0 0) 0x7faa540a9be0 con 0x7faa541440e0 2024-03-25T19:43:54.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 >> v1:172.21.15.195:6789/0 conn(0x7faa540ad700 legacy=0x7faa541e74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 >> v1:172.21.15.107:6789/0 conn(0x7faa54061830 legacy=0x7faa541df160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa3bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa3bfff640 1 monclient: found mon.c 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa3bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa540a9600 con 0x7faa541440e0 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa3bfff640 10 monclient: _finish_auth 0 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa3bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa59ed6640 5 monclient: authenticate success, global_id 4592 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa59ed6640 10 monclient: _renew_subs 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa59ed6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa59ed6640 1 -- 172.21.15.107:0/1014224940 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa540aa9d0 con 0x7faa541440e0 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa59ed6640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa59ed6640 10 monclient: _renew_subs 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa59ed6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa59ed6640 1 -- 172.21.15.107:0/1014224940 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5413f470 con 0x7faa541440e0 2024-03-25T19:43:54.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.979+0000 7faa59ed6640 1 librados: init done 2024-03-25T19:43:54.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.980+0000 7faa59ed6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:54.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.980+0000 7faa59ed6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:54.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.980+0000 7faa59ed6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:54.981 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:54.980+0000 7faa59ed6640 1 -- 172.21.15.107:0/1014224940 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa540627b0 con 0x7faa541440e0 2024-03-25T19:43:55.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.283+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa48001c50 con 0x7faa541440e0 2024-03-25T19:43:55.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.283+0000 7faa3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:55.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.283+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7faa48003a60 con 0x7faa541440e0 2024-03-25T19:43:55.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.283+0000 7faa3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:55.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.283+0000 7faa3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:55.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.283+0000 7faa3bfff640 10 monclient: dump: 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.283+0000 7faa51ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:55.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.283+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7faa48003c80 con 0x7faa541440e0 2024-03-25T19:43:55.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.284+0000 7faa537fe640 10 monclient: get_auth_request con 0x7faa4006e0f0 auth_method 0 2024-03-25T19:43:55.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.284+0000 7faa3bfff640 10 client.4592.objecter ms_handle_connect 0x7faa4006e0f0 2024-03-25T19:43:55.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.290+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7faa48054840 con 0x7faa541440e0 2024-03-25T19:43:55.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.290+0000 7faa3bfff640 10 client.4592.objecter ms_dispatch 0x7faa54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:55.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.290+0000 7faa3bfff640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:55.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.290+0000 7faa3bfff640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:55.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.290+0000 7faa3bfff640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-25T19:43:55.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.290+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa48050350 con 0x7faa541440e0 2024-03-25T19:43:55.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.290+0000 7faa3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:55.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.290+0000 7faa3bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:55.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.530+0000 7faa59ed6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:55.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.530+0000 7faa59ed6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:55.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.530+0000 7faa59ed6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:43:55.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.530+0000 7faa59ed6640 1 -- 172.21.15.107:0/1014224940 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa540a9d70 con 0x7faa541440e0 2024-03-25T19:43:55.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.558+0000 7faa3bfff640 1 -- 172.21.15.107:0/1014224940 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7faa48052280 con 0x7faa541440e0 2024-03-25T19:43:55.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.558+0000 7faa3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:55.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.558+0000 7faa3bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:55.558 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:43:55.558 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:55.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 10 librados: watch_flush enter 2024-03-25T19:43:55.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 10 librados: watch_flush exit 2024-03-25T19:43:55.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-25T19:43:55.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 10 client.4592.objecter successfully canceled tick 2024-03-25T19:43:55.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 1 -- 172.21.15.107:0/1014224940 >> v1:172.21.15.195:6800/2984408421 conn(0x7faa4006e0f0 legacy=0x7faa400705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:55.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 10 monclient: shutdown 2024-03-25T19:43:55.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:55.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 1 -- 172.21.15.107:0/1014224940 >> v1:172.21.15.107:6790/0 conn(0x7faa541440e0 legacy=0x7faa5413ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:55.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 1 -- 172.21.15.107:0/1014224940 shutdown_connections 2024-03-25T19:43:55.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 1 -- 172.21.15.107:0/1014224940 >> 172.21.15.107:0/1014224940 conn(0x7faa540b0ef0 msgr2=0x7faa540b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:55.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 1 -- 172.21.15.107:0/1014224940 shutdown_connections 2024-03-25T19:43:55.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.560+0000 7faa59ed6640 1 -- 172.21.15.107:0/1014224940 wait complete. 2024-03-25T19:43:55.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:43:55.561+0000 7faa59ed6640 1 librados: shutdown 2024-03-25T19:43:55.573 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:02.380 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:02.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.496+0000 7fd0846e2640 10 monclient: get_monmap_and_config 2024-03-25T19:44:02.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.496+0000 7fd0846e2640 10 monclient: build_initial_monmap 2024-03-25T19:44:02.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.497+0000 7fd0846e2640 10 monclient: monmap: 2024-03-25T19:44:02.497 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:02.497 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:02.497 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:44:02.497948+0000 2024-03-25T19:44:02.497 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:44:02.497948+0000 2024-03-25T19:44:02.497 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:02.497 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:02.497 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:02.497 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:44:02.497 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:44:02.497 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:02.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.497+0000 7fd0846e2640 1 Processor -- start 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.497+0000 7fd0846e2640 1 -- start start 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.497+0000 7fd0846e2640 10 monclient: init 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.498+0000 7fd0846e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.498+0000 7fd0846e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.498+0000 7fd0846e2640 10 monclient(hunting): picked mon.noname-c con 0x7fd07c06ab80 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.498+0000 7fd0846e2640 10 monclient(hunting): picked mon.noname-a con 0x7fd07c06af60 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.498+0000 7fd0846e2640 10 monclient(hunting): picked mon.noname-b con 0x7fd07c0b00b0 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.498+0000 7fd0846e2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c0b0500 con 0x7fd07c06af60 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.498+0000 7fd0846e2640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c13ffa0 con 0x7fd07c06ab80 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.498+0000 7fd0846e2640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c0af520 con 0x7fd07c0b00b0 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.498+0000 7fd0846e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:02.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.498+0000 7fd0846e2640 10 monclient(hunting): authenticate will time out at 5034.839583s 2024-03-25T19:44:02.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.510+0000 7fd08247e640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd07c06ab80 0x7fd07c0ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:58802/0 (socket says 172.21.15.107:58802) 2024-03-25T19:44:02.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.510+0000 7fd08247e640 1 -- 172.21.15.107:0/3533517561 learned_addr learned my addr 172.21.15.107:0/3533517561 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:02.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.519+0000 7fd08147c640 1 -- 172.21.15.107:0/3533517561 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060974685 0 0) 0x7fd07c13ffa0 con 0x7fd07c06ab80 2024-03-25T19:44:02.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.519+0000 7fd08147c640 10 monclient(hunting): _negotiate 2024-03-25T19:44:02.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.519+0000 7fd08147c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:02.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.519+0000 7fd08147c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:02.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.519+0000 7fd08147c640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:44:02.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.519+0000 7fd08147c640 1 -- 172.21.15.107:0/3533517561 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd068003520 con 0x7fd07c06ab80 2024-03-25T19:44:02.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.525+0000 7fd08147c640 1 -- 172.21.15.107:0/3533517561 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167599501 0 0) 0x7fd07c0b0500 con 0x7fd07c06af60 2024-03-25T19:44:02.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.525+0000 7fd08147c640 10 monclient(hunting): _negotiate 2024-03-25T19:44:02.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.525+0000 7fd08147c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:02.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.525+0000 7fd08147c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:02.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.525+0000 7fd08147c640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:44:02.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.525+0000 7fd08147c640 1 -- 172.21.15.107:0/3533517561 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd07c13ffa0 con 0x7fd07c06af60 2024-03-25T19:44:02.533 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 1 -- 172.21.15.107:0/3533517561 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3633628400 0 0) 0x7fd068003520 con 0x7fd07c06ab80 2024-03-25T19:44:02.533 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 1 -- 172.21.15.107:0/3533517561 >> v1:172.21.15.107:6790/0 conn(0x7fd07c0b00b0 legacy=0x7fd07c14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.533 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 1 -- 172.21.15.107:0/3533517561 >> v1:172.21.15.107:6789/0 conn(0x7fd07c06af60 legacy=0x7fd07c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.533 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 10 monclient: _finish_hunting 0 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 1 monclient: found mon.noname-c 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 1 -- 172.21.15.107:0/3533517561 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c072890 con 0x7fd07c06ab80 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 10 monclient: _finish_auth 0 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 10 monclient: _check_auth_tickets 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 1 -- 172.21.15.107:0/3533517561 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd078001690 con 0x7fd07c06ab80 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd0846e2640 5 monclient: authenticate success, global_id 4582 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd0846e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd08147c640 10 monclient: dump: 2024-03-25T19:44:02.534 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:02.535 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:02.535 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:02.535 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:02.535 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:02.535 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:02.535 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:02.535 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:02.535 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:02.535 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:02.535 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.533+0000 7fd0846e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:02.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.540+0000 7fd08147c640 1 -- 172.21.15.107:0/3533517561 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0780017f0 con 0x7fd07c06ab80 2024-03-25T19:44:02.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.540+0000 7fd08147c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:02.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.540+0000 7fd08147c640 1 -- 172.21.15.107:0/3533517561 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd078001c00 con 0x7fd07c06ab80 2024-03-25T19:44:02.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.540+0000 7fd08147c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:02.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.540+0000 7fd08147c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.540+0000 7fd08147c640 10 monclient: dump: 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.540+0000 7fd0846e2640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:02.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.540+0000 7fd0846e2640 4 set_mon_vals no callback set 2024-03-25T19:44:02.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.541+0000 7fd0846e2640 10 monclient: shutdown 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.541+0000 7fd0846e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.541+0000 7fd0846e2640 1 -- 172.21.15.107:0/3533517561 >> v1:172.21.15.195:6789/0 conn(0x7fd07c06ab80 legacy=0x7fd07c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.541+0000 7fd0846e2640 1 -- 172.21.15.107:0/3533517561 shutdown_connections 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.541+0000 7fd0846e2640 1 -- 172.21.15.107:0/3533517561 >> 172.21.15.107:0/3533517561 conn(0x7fd07c0b4a80 msgr2=0x7fd07c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.541+0000 7fd0846e2640 1 -- 172.21.15.107:0/3533517561 shutdown_connections 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.541+0000 7fd0846e2640 1 -- 172.21.15.107:0/3533517561 wait complete. 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.542+0000 7fd0846e2640 10 monclient: build_initial_monmap 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.542+0000 7fd0846e2640 10 monclient: monmap: 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:44:02.544 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.543+0000 7fd0846e2640 1 librados: starting msgr at 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.543+0000 7fd0846e2640 1 librados: starting objecter 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.543+0000 7fd0846e2640 1 Processor -- start 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.543+0000 7fd0846e2640 1 -- start start 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.543+0000 7fd0846e2640 1 librados: setting wanted keys 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.543+0000 7fd0846e2640 1 librados: calling monclient init 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.543+0000 7fd0846e2640 10 monclient: init 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.543+0000 7fd0846e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:02.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.543+0000 7fd0846e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:02.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.543+0000 7fd0846e2640 10 monclient(hunting): picked mon.noname-a con 0x7fd07c06ab80 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:02.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.543+0000 7fd0846e2640 10 monclient(hunting): picked mon.noname-b con 0x7fd07c0b00b0 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:02.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.544+0000 7fd0846e2640 10 monclient(hunting): picked mon.noname-c con 0x7fd07c13b170 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:02.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.544+0000 7fd0846e2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c11dbe0 con 0x7fd07c06ab80 2024-03-25T19:44:02.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.544+0000 7fd0846e2640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c11dec0 con 0x7fd07c0b00b0 2024-03-25T19:44:02.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.544+0000 7fd0846e2640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c13b670 con 0x7fd07c13b170 2024-03-25T19:44:02.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.544+0000 7fd0846e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:02.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.544+0000 7fd0846e2640 10 monclient(hunting): authenticate will time out at 5034.884901s 2024-03-25T19:44:02.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.554+0000 7fd082c7f640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd07c13b170 0x7fd07c1e2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48402/0 (socket says 172.21.15.107:48402) 2024-03-25T19:44:02.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.554+0000 7fd082c7f640 1 -- 172.21.15.107:0/4068186612 learned_addr learned my addr 172.21.15.107:0/4068186612 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:02.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.556+0000 7fd072ffd640 10 client.?.objecter ms_handle_connect 0x7fd07c0b00b0 2024-03-25T19:44:02.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.556+0000 7fd072ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:02.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.556+0000 7fd072ffd640 10 client.?.objecter ms_handle_connect 0x7fd07c13b170 2024-03-25T19:44:02.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.556+0000 7fd072ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:02.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.562+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674490343 0 0) 0x7fd07c11dec0 con 0x7fd07c0b00b0 2024-03-25T19:44:02.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.562+0000 7fd072ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:02.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.562+0000 7fd072ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:02.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.562+0000 7fd072ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:02.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.562+0000 7fd072ffd640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:44:02.562 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.562+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd060004eb0 con 0x7fd07c0b00b0 2024-03-25T19:44:02.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.570+0000 7fd072ffd640 10 client.?.objecter ms_handle_connect 0x7fd07c06ab80 2024-03-25T19:44:02.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.570+0000 7fd072ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:02.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.570+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833025555 0 0) 0x7fd07c13b670 con 0x7fd07c13b170 2024-03-25T19:44:02.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.570+0000 7fd072ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:02.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.570+0000 7fd072ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:02.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.570+0000 7fd072ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:02.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.570+0000 7fd072ffd640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:44:02.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.570+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd07c11dec0 con 0x7fd07c13b170 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.587+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1291945160 0 0) 0x7fd07c11dec0 con 0x7fd07c13b170 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.587+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd07c13b670 con 0x7fd07c13b170 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.587+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd074001c10 con 0x7fd07c13b170 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.587+0000 7fd072ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.587+0000 7fd072ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.587+0000 7fd072ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:02.587 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:02.588 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:02.588 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:02.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.587+0000 7fd072ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:02.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.588+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1912968066 0 0) 0x7fd060004eb0 con 0x7fd07c0b00b0 2024-03-25T19:44:02.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.588+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd07c11dec0 con 0x7fd07c0b00b0 2024-03-25T19:44:02.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.588+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd06c001930 con 0x7fd07c0b00b0 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.588+0000 7fd072ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.588+0000 7fd072ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.588+0000 7fd072ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:02.589 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:02.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.596+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12617123 0 0) 0x7fd07c11dbe0 con 0x7fd07c06ab80 2024-03-25T19:44:02.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.596+0000 7fd072ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:02.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.596+0000 7fd072ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:02.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.596+0000 7fd072ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:02.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.596+0000 7fd072ffd640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:44:02.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.596+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd060004eb0 con 0x7fd07c06ab80 2024-03-25T19:44:02.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 750035053 0 0) 0x7fd07c11dec0 con 0x7fd07c0b00b0 2024-03-25T19:44:02.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 >> v1:172.21.15.107:6790/0 conn(0x7fd07c13b170 legacy=0x7fd07c1e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 >> v1:172.21.15.107:6789/0 conn(0x7fd07c06ab80 legacy=0x7fd07c13aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd072ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:02.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd072ffd640 1 monclient: found mon.b 2024-03-25T19:44:02.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd072ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:02.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd072ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:44:02.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c146540 con 0x7fd07c0b00b0 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd072ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd072ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd072ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd0846e2640 5 monclient: authenticate success, global_id 4585 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd0846e2640 10 monclient: _renew_subs 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd0846e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd0846e2640 1 -- 172.21.15.107:0/4068186612 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c145fc0 con 0x7fd07c0b00b0 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd0846e2640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd0846e2640 10 monclient: _renew_subs 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd0846e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd0846e2640 1 -- 172.21.15.107:0/4068186612 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c146310 con 0x7fd07c0b00b0 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.601+0000 7fd0846e2640 1 librados: init done 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.602+0000 7fd0846e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.602+0000 7fd0846e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:02.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.602+0000 7fd0846e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:44:02.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.602+0000 7fd0846e2640 1 -- 172.21.15.107:0/4068186612 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07c11cc70 con 0x7fd07c0b00b0 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.885+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd06c0035f0 con 0x7fd07c0b00b0 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.885+0000 7fd072ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.885+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd06c003900 con 0x7fd07c0b00b0 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.885+0000 7fd072ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.885+0000 7fd072ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.885+0000 7fd072ffd640 10 monclient: dump: 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:02.885 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:02.886 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:02.886 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:02.886 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:02.886 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:02.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.885+0000 7fd080c7b640 4 set_mon_vals no callback set 2024-03-25T19:44:02.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.886+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fd06c019910 con 0x7fd07c0b00b0 2024-03-25T19:44:02.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.887+0000 7fd08247e640 10 monclient: get_auth_request con 0x7fd06006dee0 auth_method 0 2024-03-25T19:44:02.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.887+0000 7fd072ffd640 10 client.4585.objecter ms_handle_connect 0x7fd06006dee0 2024-03-25T19:44:02.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.892+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fd06c053660 con 0x7fd07c0b00b0 2024-03-25T19:44:02.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.892+0000 7fd072ffd640 10 client.4585.objecter ms_dispatch 0x7fd07c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:02.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.892+0000 7fd072ffd640 3 client.4585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:02.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.892+0000 7fd072ffd640 3 client.4585.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:02.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.892+0000 7fd072ffd640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-25T19:44:02.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.892+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd06c04efa0 con 0x7fd07c0b00b0 2024-03-25T19:44:02.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.892+0000 7fd072ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:02.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:02.892+0000 7fd072ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:03.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.150+0000 7fd0846e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:03.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.150+0000 7fd0846e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:03.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.150+0000 7fd0846e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:44:03.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.150+0000 7fd0846e2640 1 -- 172.21.15.107:0/4068186612 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd07c0627b0 con 0x7fd07c0b00b0 2024-03-25T19:44:03.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.185+0000 7fd072ffd640 1 -- 172.21.15.107:0/4068186612 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fd06c0510a0 con 0x7fd07c0b00b0 2024-03-25T19:44:03.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.185+0000 7fd072ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:03.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.185+0000 7fd072ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:03.185 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:44:03.185 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:03.187 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 10 librados: watch_flush enter 2024-03-25T19:44:03.187 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 10 librados: watch_flush exit 2024-03-25T19:44:03.187 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-25T19:44:03.187 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 10 client.4585.objecter successfully canceled tick 2024-03-25T19:44:03.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 1 -- 172.21.15.107:0/4068186612 >> v1:172.21.15.195:6800/2984408421 conn(0x7fd06006dee0 legacy=0x7fd0600703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:03.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 10 monclient: shutdown 2024-03-25T19:44:03.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:03.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 1 -- 172.21.15.107:0/4068186612 >> v1:172.21.15.195:6789/0 conn(0x7fd07c0b00b0 legacy=0x7fd07c1d7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:03.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 1 -- 172.21.15.107:0/4068186612 shutdown_connections 2024-03-25T19:44:03.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 1 -- 172.21.15.107:0/4068186612 >> 172.21.15.107:0/4068186612 conn(0x7fd07c0b4a80 msgr2=0x7fd07c13f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:03.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 1 -- 172.21.15.107:0/4068186612 shutdown_connections 2024-03-25T19:44:03.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 1 -- 172.21.15.107:0/4068186612 wait complete. 2024-03-25T19:44:03.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:03.187+0000 7fd0846e2640 1 librados: shutdown 2024-03-25T19:44:03.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:05.804 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:08:36.119 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:05.820 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:08:36.137 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:05.852 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:08:36.168 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:05.870 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:08:36.186 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:05.873 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:08:36.159 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:05.878 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:08:36.165 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:05.879 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:08:36.196 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:05.906 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:08:36.193 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:05.915 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:08:36.231 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:10.006 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:10.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.124+0000 7fddc159d640 10 monclient: get_monmap_and_config 2024-03-25T19:44:10.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.124+0000 7fddc159d640 10 monclient: build_initial_monmap 2024-03-25T19:44:10.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.124+0000 7fddc159d640 10 monclient: monmap: 2024-03-25T19:44:10.125 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:10.125 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:10.125 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:44:10.125916+0000 2024-03-25T19:44:10.125 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:44:10.125916+0000 2024-03-25T19:44:10.125 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:10.125 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:10.125 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:10.125 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:44:10.125 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:44:10.125 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:10.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.125+0000 7fddc159d640 1 Processor -- start 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.125+0000 7fddc159d640 1 -- start start 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.125+0000 7fddc159d640 10 monclient: init 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.126+0000 7fddc159d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.126+0000 7fddc159d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.126+0000 7fddc159d640 10 monclient(hunting): picked mon.noname-a con 0x7fddbc0a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.126+0000 7fddc159d640 10 monclient(hunting): picked mon.noname-c con 0x7fddbc13f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.126+0000 7fddc159d640 10 monclient(hunting): picked mon.noname-b con 0x7fddbc1430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.126+0000 7fddc159d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc142520 con 0x7fddbc0a98f0 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.126+0000 7fddc159d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc0b28a0 con 0x7fddbc13f710 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.126+0000 7fddc159d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc0b2a30 con 0x7fddbc1430c0 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.126+0000 7fddc159d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:10.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.126+0000 7fddc159d640 10 monclient(hunting): authenticate will time out at 5042.467705s 2024-03-25T19:44:10.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.132+0000 7fddbaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a98f0 0x7fddbc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45726/0 (socket says 172.21.15.107:45726) 2024-03-25T19:44:10.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.132+0000 7fddbaffd640 1 -- 172.21.15.107:0/833003966 learned_addr learned my addr 172.21.15.107:0/833003966 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:10.194 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.194+0000 7fddb9ffb640 1 -- 172.21.15.107:0/833003966 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128714894 0 0) 0x7fddbc0b2a30 con 0x7fddbc1430c0 2024-03-25T19:44:10.194 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.195+0000 7fddb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:10.194 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.195+0000 7fddb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:10.195 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.195+0000 7fddb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:10.195 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.195+0000 7fddb9ffb640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:44:10.195 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.195+0000 7fddb9ffb640 1 -- 172.21.15.107:0/833003966 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda4003520 con 0x7fddbc1430c0 2024-03-25T19:44:10.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 1 -- 172.21.15.107:0/833003966 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2280044813 0 0) 0x7fdda4003520 con 0x7fddbc1430c0 2024-03-25T19:44:10.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 1 -- 172.21.15.107:0/833003966 >> v1:172.21.15.195:6789/0 conn(0x7fddbc13f710 legacy=0x7fddbc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 1 -- 172.21.15.107:0/833003966 >> v1:172.21.15.107:6789/0 conn(0x7fddbc0a98f0 legacy=0x7fddbc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:10.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 1 monclient: found mon.noname-b 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 1 -- 172.21.15.107:0/833003966 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc0b44c0 con 0x7fddbc1430c0 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 1 -- 172.21.15.107:0/833003966 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fddb4001a10 con 0x7fddbc1430c0 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddb9ffb640 10 monclient: dump: 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:10.207 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:10.208 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:10.208 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:10.208 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:10.208 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:10.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddc159d640 5 monclient: authenticate success, global_id 4601 2024-03-25T19:44:10.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.206+0000 7fddc159d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:10.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.209+0000 7fddb9ffb640 1 -- 172.21.15.107:0/833003966 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddb4001b70 con 0x7fddbc1430c0 2024-03-25T19:44:10.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.209+0000 7fddb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:10.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.209+0000 7fddc159d640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:10.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.209+0000 7fddc159d640 4 set_mon_vals no callback set 2024-03-25T19:44:10.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.210+0000 7fddc159d640 10 monclient: shutdown 2024-03-25T19:44:10.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.210+0000 7fddc159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:10.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.210+0000 7fddc159d640 1 -- 172.21.15.107:0/833003966 >> v1:172.21.15.107:6790/0 conn(0x7fddbc1430c0 legacy=0x7fddbc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.210+0000 7fddc159d640 1 -- 172.21.15.107:0/833003966 shutdown_connections 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.210+0000 7fddc159d640 1 -- 172.21.15.107:0/833003966 >> 172.21.15.107:0/833003966 conn(0x7fddbc0b7ea0 msgr2=0x7fddbc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.210+0000 7fddc159d640 1 -- 172.21.15.107:0/833003966 shutdown_connections 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.210+0000 7fddc159d640 1 -- 172.21.15.107:0/833003966 wait complete. 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.211+0000 7fddc159d640 10 monclient: build_initial_monmap 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.211+0000 7fddc159d640 10 monclient: monmap: 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:10.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.211+0000 7fddc159d640 1 librados: starting msgr at 2024-03-25T19:44:10.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.211+0000 7fddc159d640 1 librados: starting objecter 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.211+0000 7fddc159d640 1 Processor -- start 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.211+0000 7fddc159d640 1 -- start start 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.211+0000 7fddc159d640 1 librados: setting wanted keys 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.211+0000 7fddc159d640 1 librados: calling monclient init 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.211+0000 7fddc159d640 10 monclient: init 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.212+0000 7fddc159d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.212+0000 7fddc159d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.212+0000 7fddc159d640 10 monclient(hunting): picked mon.noname-c con 0x7fddbc0a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.212+0000 7fddc159d640 10 monclient(hunting): picked mon.noname-a con 0x7fddbc13f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.212+0000 7fddc159d640 10 monclient(hunting): picked mon.noname-b con 0x7fddbc1430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.212+0000 7fddc159d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc139a00 con 0x7fddbc13f710 2024-03-25T19:44:10.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.212+0000 7fddc159d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc1357f0 con 0x7fddbc1430c0 2024-03-25T19:44:10.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.212+0000 7fddc159d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc135360 con 0x7fddbc0a98f0 2024-03-25T19:44:10.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.212+0000 7fddc159d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:10.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.212+0000 7fddc159d640 10 monclient(hunting): authenticate will time out at 5042.553653s 2024-03-25T19:44:10.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.214+0000 7fddbb7fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fddbc1430c0 0x7fddbc136420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:34914/0 (socket says 172.21.15.107:34914) 2024-03-25T19:44:10.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.214+0000 7fddbb7fe640 1 -- 172.21.15.107:0/1782268093 learned_addr learned my addr 172.21.15.107:0/1782268093 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:10.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.219+0000 7fdda37fe640 10 client.?.objecter ms_handle_connect 0x7fddbc0a98f0 2024-03-25T19:44:10.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.219+0000 7fdda37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:10.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.230+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736469119 0 0) 0x7fddbc135360 con 0x7fddbc0a98f0 2024-03-25T19:44:10.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.230+0000 7fdda37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:10.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.230+0000 7fdda37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:10.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.230+0000 7fdda37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:10.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.230+0000 7fdda37fe640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:44:10.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.230+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda8004eb0 con 0x7fddbc0a98f0 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.248+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1924358729 0 0) 0x7fdda8004eb0 con 0x7fddbc0a98f0 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.248+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddbc135360 con 0x7fddbc0a98f0 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.248+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fddac0018a0 con 0x7fddbc0a98f0 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.248+0000 7fdda37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.248+0000 7fdda37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.248+0000 7fdda37fe640 10 monclient(hunting): dump: 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:10.248 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:10.249 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:10.249 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:10.249 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:10.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.248+0000 7fdda37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:10.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.260+0000 7fdda37fe640 10 client.?.objecter ms_handle_connect 0x7fddbc13f710 2024-03-25T19:44:10.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.260+0000 7fdda37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:10.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.268+0000 7fdda37fe640 10 client.?.objecter ms_handle_connect 0x7fddbc1430c0 2024-03-25T19:44:10.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.268+0000 7fdda37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:10.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.268+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334598013 0 0) 0x7fddbc139a00 con 0x7fddbc13f710 2024-03-25T19:44:10.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.268+0000 7fdda37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:10.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.268+0000 7fdda37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:10.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.268+0000 7fdda37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:10.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.268+0000 7fdda37fe640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:44:10.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.268+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda8004eb0 con 0x7fddbc13f710 2024-03-25T19:44:10.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2670464818 0 0) 0x7fddbc135360 con 0x7fddbc0a98f0 2024-03-25T19:44:10.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 >> v1:172.21.15.195:6789/0 conn(0x7fddbc1430c0 legacy=0x7fddbc136420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 >> v1:172.21.15.107:6789/0 conn(0x7fddbc13f710 legacy=0x7fddbc135d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fdda37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:10.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fdda37fe640 1 monclient: found mon.c 2024-03-25T19:44:10.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fdda37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:10.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fdda37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:44:10.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc1354f0 con 0x7fddbc0a98f0 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fdda37fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fdda37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fdda37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fddc159d640 5 monclient: authenticate success, global_id 4604 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fddc159d640 10 monclient: _renew_subs 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fddc159d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fddc159d640 1 -- 172.21.15.107:0/1782268093 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddbc137dd0 con 0x7fddbc0a98f0 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fddc159d640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fddc159d640 10 monclient: _renew_subs 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fddc159d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fddc159d640 1 -- 172.21.15.107:0/1782268093 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc134720 con 0x7fddbc0a98f0 2024-03-25T19:44:10.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.270+0000 7fddc159d640 1 librados: init done 2024-03-25T19:44:10.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.271+0000 7fddc159d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:10.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.271+0000 7fddc159d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:10.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.271+0000 7fddc159d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:44:10.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.271+0000 7fddc159d640 1 -- 172.21.15.107:0/1782268093 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddbc135980 con 0x7fddbc0a98f0 2024-03-25T19:44:10.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.596+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddac0047e0 con 0x7fddbc0a98f0 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.596+0000 7fdda37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.596+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fddac004c30 con 0x7fddbc0a98f0 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.596+0000 7fdda37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.596+0000 7fdda37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.596+0000 7fdda37fe640 10 monclient: dump: 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:10.596 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:10.597 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:10.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.596+0000 7fddb97fa640 4 set_mon_vals no callback set 2024-03-25T19:44:10.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.597+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fddac004e50 con 0x7fddbc0a98f0 2024-03-25T19:44:10.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.597+0000 7fddba7fc640 10 monclient: get_auth_request con 0x7fdda806dd40 auth_method 0 2024-03-25T19:44:10.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.598+0000 7fdda37fe640 10 client.4604.objecter ms_handle_connect 0x7fdda806dd40 2024-03-25T19:44:10.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.605+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fddac054800 con 0x7fddbc0a98f0 2024-03-25T19:44:10.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.605+0000 7fdda37fe640 10 client.4604.objecter ms_dispatch 0x7fddbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:10.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.605+0000 7fdda37fe640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:10.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.605+0000 7fdda37fe640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:10.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.605+0000 7fdda37fe640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T19:44:10.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.605+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fddac050310 con 0x7fddbc0a98f0 2024-03-25T19:44:10.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.605+0000 7fdda37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:10.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.605+0000 7fdda37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:10.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.850+0000 7fddc159d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:10.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.850+0000 7fddc159d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:10.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.850+0000 7fddc159d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:44:10.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.850+0000 7fddc159d640 1 -- 172.21.15.107:0/1782268093 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddbc0627b0 con 0x7fddbc0a98f0 2024-03-25T19:44:10.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.884+0000 7fdda37fe640 1 -- 172.21.15.107:0/1782268093 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fddac052240 con 0x7fddbc0a98f0 2024-03-25T19:44:10.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.884+0000 7fdda37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:10.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.884+0000 7fdda37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:10.885 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:44:10.885 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:10.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 10 librados: watch_flush enter 2024-03-25T19:44:10.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 10 librados: watch_flush exit 2024-03-25T19:44:10.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T19:44:10.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 10 client.4604.objecter successfully canceled tick 2024-03-25T19:44:10.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 1 -- 172.21.15.107:0/1782268093 >> v1:172.21.15.195:6800/2984408421 conn(0x7fdda806dd40 legacy=0x7fdda8070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 10 monclient: shutdown 2024-03-25T19:44:10.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:10.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 1 -- 172.21.15.107:0/1782268093 >> v1:172.21.15.107:6790/0 conn(0x7fddbc0a98f0 legacy=0x7fddbc134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:10.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 1 -- 172.21.15.107:0/1782268093 shutdown_connections 2024-03-25T19:44:10.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 1 -- 172.21.15.107:0/1782268093 >> 172.21.15.107:0/1782268093 conn(0x7fddbc0b7ea0 msgr2=0x7fddbc13c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:10.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 1 -- 172.21.15.107:0/1782268093 shutdown_connections 2024-03-25T19:44:10.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.887+0000 7fddc159d640 1 -- 172.21.15.107:0/1782268093 wait complete. 2024-03-25T19:44:10.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:10.888+0000 7fddc159d640 1 librados: shutdown 2024-03-25T19:44:10.900 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:17.706 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.822+0000 7fd57cda5640 10 monclient: get_monmap_and_config 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.822+0000 7fd57cda5640 10 monclient: build_initial_monmap 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.822+0000 7fd57cda5640 10 monclient: monmap: 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:44:17.823094+0000 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:44:17.823094+0000 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:44:17.822 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:44:17.823 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:17.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.823+0000 7fd57cda5640 1 Processor -- start 2024-03-25T19:44:17.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.823+0000 7fd57cda5640 1 -- start start 2024-03-25T19:44:17.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.823+0000 7fd57cda5640 10 monclient: init 2024-03-25T19:44:17.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.823+0000 7fd57cda5640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:17.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.823+0000 7fd57cda5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:17.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.823+0000 7fd57cda5640 10 monclient(hunting): picked mon.noname-c con 0x7fd5780a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:17.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.823+0000 7fd57cda5640 10 monclient(hunting): picked mon.noname-a con 0x7fd57813f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:17.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.824+0000 7fd57cda5640 10 monclient(hunting): picked mon.noname-b con 0x7fd5781430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:17.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.824+0000 7fd57cda5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd578142520 con 0x7fd57813f710 2024-03-25T19:44:17.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.824+0000 7fd57cda5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5780b28a0 con 0x7fd5780a98f0 2024-03-25T19:44:17.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.824+0000 7fd57cda5640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5780b2a30 con 0x7fd5781430c0 2024-03-25T19:44:17.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.824+0000 7fd57cda5640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:17.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.824+0000 7fd57cda5640 10 monclient(hunting): authenticate will time out at 5050.164809s 2024-03-25T19:44:17.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.832+0000 7fd576d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd5781430c0 0x7fd57814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:56058/0 (socket says 172.21.15.107:56058) 2024-03-25T19:44:17.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.832+0000 7fd576d9d640 1 -- 172.21.15.107:0/873959560 learned_addr learned my addr 172.21.15.107:0/873959560 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:17.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412906361 0 0) 0x7fd5780b2a30 con 0x7fd5781430c0 2024-03-25T19:44:17.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:17.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:17.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:17.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:44:17.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd564003520 con 0x7fd5781430c0 2024-03-25T19:44:17.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932602205 0 0) 0x7fd578142520 con 0x7fd57813f710 2024-03-25T19:44:17.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:17.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:17.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:17.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:44:17.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.844+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5780b2a30 con 0x7fd57813f710 2024-03-25T19:44:17.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.849+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428992418 0 0) 0x7fd5780b28a0 con 0x7fd5780a98f0 2024-03-25T19:44:17.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.849+0000 7fd575d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:17.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.849+0000 7fd575d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:17.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.849+0000 7fd575d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:17.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.849+0000 7fd575d9b640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:44:17.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.849+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd578142520 con 0x7fd5780a98f0 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 233597229 0 0) 0x7fd564003520 con 0x7fd5781430c0 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 >> v1:172.21.15.195:6789/0 conn(0x7fd5780a98f0 legacy=0x7fd5780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 >> v1:172.21.15.107:6789/0 conn(0x7fd57813f710 legacy=0x7fd5781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 1 monclient: found mon.noname-b 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5780b44c0 con 0x7fd5781430c0 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 10 monclient: _finish_auth 0 2024-03-25T19:44:17.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd570001a10 con 0x7fd5781430c0 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd575d9b640 10 monclient: dump: 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:17.858 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:17.859 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:17.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd57cda5640 5 monclient: authenticate success, global_id 4607 2024-03-25T19:44:17.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.857+0000 7fd57cda5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.884+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd570001b70 con 0x7fd5781430c0 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.884+0000 7fd575d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.884+0000 7fd575d9b640 1 -- 172.21.15.107:0/873959560 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd570001f80 con 0x7fd5781430c0 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.884+0000 7fd57cda5640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.884+0000 7fd57cda5640 4 set_mon_vals no callback set 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.884+0000 7fd575d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.884+0000 7fd575d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.884+0000 7fd575d9b640 10 monclient: dump: 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:17.884 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:17.885 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:17.885 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:17.885 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:17.885 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:17.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.885+0000 7fd57cda5640 10 monclient: shutdown 2024-03-25T19:44:17.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.885+0000 7fd57cda5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:17.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.885+0000 7fd57cda5640 1 -- 172.21.15.107:0/873959560 >> v1:172.21.15.107:6790/0 conn(0x7fd5781430c0 legacy=0x7fd57814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.885+0000 7fd57cda5640 1 -- 172.21.15.107:0/873959560 shutdown_connections 2024-03-25T19:44:17.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.885+0000 7fd57cda5640 1 -- 172.21.15.107:0/873959560 >> 172.21.15.107:0/873959560 conn(0x7fd5780b7ea0 msgr2=0x7fd5780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:17.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.885+0000 7fd57cda5640 1 -- 172.21.15.107:0/873959560 shutdown_connections 2024-03-25T19:44:17.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.885+0000 7fd57cda5640 1 -- 172.21.15.107:0/873959560 wait complete. 2024-03-25T19:44:17.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.886+0000 7fd57cda5640 10 monclient: build_initial_monmap 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.886+0000 7fd57cda5640 10 monclient: monmap: 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.886+0000 7fd57cda5640 1 librados: starting msgr at 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.886+0000 7fd57cda5640 1 librados: starting objecter 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.886+0000 7fd57cda5640 1 Processor -- start 2024-03-25T19:44:17.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.886+0000 7fd57cda5640 1 -- start start 2024-03-25T19:44:17.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.886+0000 7fd57cda5640 1 librados: setting wanted keys 2024-03-25T19:44:17.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.886+0000 7fd57cda5640 1 librados: calling monclient init 2024-03-25T19:44:17.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.886+0000 7fd57cda5640 10 monclient: init 2024-03-25T19:44:17.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.887+0000 7fd57cda5640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:17.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.887+0000 7fd57cda5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:17.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.887+0000 7fd57cda5640 10 monclient(hunting): picked mon.noname-c con 0x7fd5780a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:17.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.887+0000 7fd57cda5640 10 monclient(hunting): picked mon.noname-b con 0x7fd57813f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:17.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.887+0000 7fd57cda5640 10 monclient(hunting): picked mon.noname-a con 0x7fd5781430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:17.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.887+0000 7fd57cda5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5781d8340 con 0x7fd5781430c0 2024-03-25T19:44:17.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.887+0000 7fd57cda5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5781d7710 con 0x7fd57813f710 2024-03-25T19:44:17.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.887+0000 7fd57cda5640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd578138e90 con 0x7fd5780a98f0 2024-03-25T19:44:17.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.887+0000 7fd57cda5640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:17.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.887+0000 7fd57cda5640 10 monclient(hunting): authenticate will time out at 5050.228419s 2024-03-25T19:44:17.894 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.894+0000 7fd57659c640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd5780a98f0 0x7fd5781d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:56120/0 (socket says 172.21.15.107:56120) 2024-03-25T19:44:17.894 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.894+0000 7fd57659c640 1 -- 172.21.15.107:0/3548530992 learned_addr learned my addr 172.21.15.107:0/3548530992 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:17.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.897+0000 7fd56effd640 10 client.?.objecter ms_handle_connect 0x7fd5780a98f0 2024-03-25T19:44:17.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.897+0000 7fd56effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:17.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.898+0000 7fd56effd640 10 client.?.objecter ms_handle_connect 0x7fd57813f710 2024-03-25T19:44:17.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.898+0000 7fd56effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:17.903 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.904+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768396969 0 0) 0x7fd578138e90 con 0x7fd5780a98f0 2024-03-25T19:44:17.903 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.904+0000 7fd56effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:17.903 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.904+0000 7fd56effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:17.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.904+0000 7fd56effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:17.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.904+0000 7fd56effd640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:44:17.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.904+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd558004eb0 con 0x7fd5780a98f0 2024-03-25T19:44:17.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.905+0000 7fd56effd640 10 client.?.objecter ms_handle_connect 0x7fd5781430c0 2024-03-25T19:44:17.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.905+0000 7fd56effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:17.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.909+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985520568 0 0) 0x7fd5781d8340 con 0x7fd5781430c0 2024-03-25T19:44:17.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.909+0000 7fd56effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:17.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.909+0000 7fd56effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:17.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.909+0000 7fd56effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:17.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.909+0000 7fd56effd640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:44:17.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.909+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd578138e90 con 0x7fd5781430c0 2024-03-25T19:44:17.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.916+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262092329 0 0) 0x7fd5781d7710 con 0x7fd57813f710 2024-03-25T19:44:17.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.916+0000 7fd56effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:17.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.916+0000 7fd56effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:17.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.916+0000 7fd56effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:17.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.916+0000 7fd56effd640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:44:17.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.916+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5781d8340 con 0x7fd57813f710 2024-03-25T19:44:17.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 741984099 0 0) 0x7fd558004eb0 con 0x7fd5780a98f0 2024-03-25T19:44:17.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5781d7710 con 0x7fd5780a98f0 2024-03-25T19:44:17.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd568001a10 con 0x7fd5780a98f0 2024-03-25T19:44:17.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:17.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:17.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 10 monclient(hunting): dump: 2024-03-25T19:44:17.926 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:17.926 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1517093324 0 0) 0x7fd578138e90 con 0x7fd5781430c0 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd558004eb0 con 0x7fd5781430c0 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd570001e80 con 0x7fd5781430c0 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:17.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.926+0000 7fd56effd640 10 monclient(hunting): dump: 2024-03-25T19:44:17.928 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:17.928 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:17.928 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:17.928 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:17.928 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:17.928 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:17.928 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:17.928 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:17.928 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:17.928 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:17.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.931+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1309734452 0 0) 0x7fd558004eb0 con 0x7fd5781430c0 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.931+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 >> v1:172.21.15.107:6790/0 conn(0x7fd5780a98f0 legacy=0x7fd5781d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.931+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 >> v1:172.21.15.195:6789/0 conn(0x7fd57813f710 legacy=0x7fd578138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.931+0000 7fd56effd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.931+0000 7fd56effd640 1 monclient: found mon.a 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.931+0000 7fd56effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.931+0000 7fd56effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.931+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd578139020 con 0x7fd5781430c0 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.931+0000 7fd56effd640 10 monclient: _finish_auth 0 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.931+0000 7fd56effd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.931+0000 7fd56effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.932+0000 7fd57cda5640 5 monclient: authenticate success, global_id 4605 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.932+0000 7fd57cda5640 10 monclient: _renew_subs 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.932+0000 7fd57cda5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.932+0000 7fd57cda5640 1 -- 172.21.15.107:0/3548530992 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd578137f30 con 0x7fd5781430c0 2024-03-25T19:44:17.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.932+0000 7fd57cda5640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:17.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.932+0000 7fd57cda5640 10 monclient: _renew_subs 2024-03-25T19:44:17.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.932+0000 7fd57cda5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:17.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.932+0000 7fd57cda5640 1 -- 172.21.15.107:0/3548530992 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5781352b0 con 0x7fd5781430c0 2024-03-25T19:44:17.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.932+0000 7fd57cda5640 1 librados: init done 2024-03-25T19:44:17.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.933+0000 7fd57cda5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:17.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.933+0000 7fd57cda5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:17.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.933+0000 7fd57cda5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:17.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:17.933+0000 7fd57cda5640 1 -- 172.21.15.107:0/3548530992 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5781d7500 con 0x7fd5781430c0 2024-03-25T19:44:18.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.237+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd570001fe0 con 0x7fd5781430c0 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.237+0000 7fd56effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.237+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fd570004fb0 con 0x7fd5781430c0 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.237+0000 7fd56effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.237+0000 7fd56effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.237+0000 7fd56effd640 10 monclient: dump: 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:18.237 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:18.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.237+0000 7fd57559a640 4 set_mon_vals no callback set 2024-03-25T19:44:18.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.237+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fd57001af80 con 0x7fd5781430c0 2024-03-25T19:44:18.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.238+0000 7fd57659c640 10 monclient: get_auth_request con 0x7fd55806df70 auth_method 0 2024-03-25T19:44:18.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.238+0000 7fd56effd640 10 client.4605.objecter ms_handle_connect 0x7fd55806df70 2024-03-25T19:44:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.243+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fd570054ad0 con 0x7fd5781430c0 2024-03-25T19:44:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.243+0000 7fd56effd640 10 client.4605.objecter ms_dispatch 0x7fd578000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.243+0000 7fd56effd640 3 client.4605.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.243+0000 7fd56effd640 3 client.4605.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.243+0000 7fd56effd640 20 client.4605.objecter dump_active .. 0 homeless 2024-03-25T19:44:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.243+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5700505e0 con 0x7fd5781430c0 2024-03-25T19:44:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.243+0000 7fd56effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.243+0000 7fd56effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:18.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.472+0000 7fd57cda5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:18.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.472+0000 7fd57cda5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:18.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.472+0000 7fd57cda5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:18.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.472+0000 7fd57cda5640 1 -- 172.21.15.107:0/3548530992 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5780627b0 con 0x7fd5781430c0 2024-03-25T19:44:18.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.497+0000 7fd56effd640 1 -- 172.21.15.107:0/3548530992 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fd570052510 con 0x7fd5781430c0 2024-03-25T19:44:18.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.497+0000 7fd56effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:18.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.497+0000 7fd56effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:18.497 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:44:18.497 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:18.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.499+0000 7fd57cda5640 10 librados: watch_flush enter 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.499+0000 7fd57cda5640 10 librados: watch_flush exit 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.499+0000 7fd57cda5640 20 client.4605.objecter shutdown clearing up homeless session... 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.499+0000 7fd57cda5640 10 client.4605.objecter successfully canceled tick 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.499+0000 7fd57cda5640 1 -- 172.21.15.107:0/3548530992 >> v1:172.21.15.195:6800/2984408421 conn(0x7fd55806df70 legacy=0x7fd558070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.499+0000 7fd57cda5640 10 monclient: shutdown 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.499+0000 7fd57cda5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.499+0000 7fd57cda5640 1 -- 172.21.15.107:0/3548530992 >> v1:172.21.15.107:6789/0 conn(0x7fd5781430c0 legacy=0x7fd578133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.499+0000 7fd57cda5640 1 -- 172.21.15.107:0/3548530992 shutdown_connections 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.499+0000 7fd57cda5640 1 -- 172.21.15.107:0/3548530992 >> 172.21.15.107:0/3548530992 conn(0x7fd5780b7ea0 msgr2=0x7fd578074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.499+0000 7fd57cda5640 1 -- 172.21.15.107:0/3548530992 shutdown_connections 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.500+0000 7fd57cda5640 1 -- 172.21.15.107:0/3548530992 wait complete. 2024-03-25T19:44:18.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:18.500+0000 7fd57cda5640 1 librados: shutdown 2024-03-25T19:44:18.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:25.319 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:25.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.438+0000 7f2d5ce0f640 10 monclient: get_monmap_and_config 2024-03-25T19:44:25.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.438+0000 7f2d5ce0f640 10 monclient: build_initial_monmap 2024-03-25T19:44:25.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.438+0000 7f2d5ce0f640 10 monclient: monmap: 2024-03-25T19:44:25.438 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:25.438 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:25.438 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:44:25.439402+0000 2024-03-25T19:44:25.438 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:44:25.439402+0000 2024-03-25T19:44:25.438 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:25.438 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:25.438 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:25.439 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:44:25.439 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:44:25.439 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.439+0000 7f2d5ce0f640 1 Processor -- start 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.439+0000 7f2d5ce0f640 1 -- start start 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.439+0000 7f2d5ce0f640 10 monclient: init 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.439+0000 7f2d5ce0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.439+0000 7f2d5ce0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.439+0000 7f2d5ce0f640 10 monclient(hunting): picked mon.noname-a con 0x7f2d580b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.439+0000 7f2d5ce0f640 10 monclient(hunting): picked mon.noname-b con 0x7f2d580b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.440+0000 7f2d5ce0f640 10 monclient(hunting): picked mon.noname-c con 0x7f2d5814be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.440+0000 7f2d5ce0f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d580af940 con 0x7f2d580b4d60 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.440+0000 7f2d5ce0f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d580b0a70 con 0x7f2d5814be90 2024-03-25T19:44:25.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.440+0000 7f2d5ce0f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d580b0d50 con 0x7f2d580b1730 2024-03-25T19:44:25.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.440+0000 7f2d5ce0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:25.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.440+0000 7f2d5ce0f640 10 monclient(hunting): authenticate will time out at 5057.780796s 2024-03-25T19:44:25.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.444+0000 7f2d5659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2d580b4d60 0x7f2d5813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35566/0 (socket says 172.21.15.107:35566) 2024-03-25T19:44:25.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.444+0000 7f2d5659c640 1 -- 172.21.15.107:0/2644719698 learned_addr learned my addr 172.21.15.107:0/2644719698 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.461+0000 7f2d55d9b640 1 -- 172.21.15.107:0/2644719698 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368270645 0 0) 0x7f2d580b0a70 con 0x7f2d5814be90 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.461+0000 7f2d55d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.461+0000 7f2d55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.461+0000 7f2d55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.461+0000 7f2d55d9b640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:44:25.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.461+0000 7f2d55d9b640 1 -- 172.21.15.107:0/2644719698 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d44003520 con 0x7f2d5814be90 2024-03-25T19:44:25.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.465+0000 7f2d55d9b640 1 -- 172.21.15.107:0/2644719698 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488216558 0 0) 0x7f2d580af940 con 0x7f2d580b4d60 2024-03-25T19:44:25.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.465+0000 7f2d55d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:25.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.465+0000 7f2d55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:25.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.465+0000 7f2d55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:25.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.465+0000 7f2d55d9b640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:44:25.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.465+0000 7f2d55d9b640 1 -- 172.21.15.107:0/2644719698 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d580b0a70 con 0x7f2d580b4d60 2024-03-25T19:44:25.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 1 -- 172.21.15.107:0/2644719698 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2313409128 0 0) 0x7f2d44003520 con 0x7f2d5814be90 2024-03-25T19:44:25.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 1 -- 172.21.15.107:0/2644719698 >> v1:172.21.15.107:6790/0 conn(0x7f2d580b1730 legacy=0x7f2d580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 1 -- 172.21.15.107:0/2644719698 >> v1:172.21.15.107:6789/0 conn(0x7f2d580b4d60 legacy=0x7f2d5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 1 monclient: found mon.noname-c 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 1 -- 172.21.15.107:0/2644719698 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d580b1030 con 0x7f2d5814be90 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 10 monclient: _finish_auth 0 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 1 -- 172.21.15.107:0/2644719698 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2d50001a10 con 0x7f2d5814be90 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d55d9b640 10 monclient: dump: 2024-03-25T19:44:25.473 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:25.474 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:25.474 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:25.474 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:25.474 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:25.474 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:25.474 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:25.474 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:25.474 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:25.474 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:25.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d5ce0f640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:44:25.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.473+0000 7f2d5ce0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:25.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.489+0000 7f2d55d9b640 1 -- 172.21.15.107:0/2644719698 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d50001b70 con 0x7f2d5814be90 2024-03-25T19:44:25.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.489+0000 7f2d55d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:25.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.489+0000 7f2d55d9b640 1 -- 172.21.15.107:0/2644719698 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2d50001f80 con 0x7f2d5814be90 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.489+0000 7f2d55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.489+0000 7f2d55d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.489+0000 7f2d55d9b640 10 monclient: dump: 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.489+0000 7f2d5ce0f640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:25.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.489+0000 7f2d5ce0f640 4 set_mon_vals no callback set 2024-03-25T19:44:25.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.489+0000 7f2d5ce0f640 10 monclient: shutdown 2024-03-25T19:44:25.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.490+0000 7f2d5ce0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:25.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.490+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2644719698 >> v1:172.21.15.195:6789/0 conn(0x7f2d5814be90 legacy=0x7f2d5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:25.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.490+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2644719698 shutdown_connections 2024-03-25T19:44:25.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.490+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2644719698 >> 172.21.15.107:0/2644719698 conn(0x7f2d580617a0 msgr2=0x7f2d58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:25.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.490+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2644719698 shutdown_connections 2024-03-25T19:44:25.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.490+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2644719698 wait complete. 2024-03-25T19:44:25.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.491+0000 7f2d5ce0f640 10 monclient: build_initial_monmap 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.491+0000 7f2d5ce0f640 10 monclient: monmap: 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.491+0000 7f2d5ce0f640 1 librados: starting msgr at 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.491+0000 7f2d5ce0f640 1 librados: starting objecter 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.491+0000 7f2d5ce0f640 1 Processor -- start 2024-03-25T19:44:25.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.491+0000 7f2d5ce0f640 1 -- start start 2024-03-25T19:44:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.491+0000 7f2d5ce0f640 1 librados: setting wanted keys 2024-03-25T19:44:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.491+0000 7f2d5ce0f640 1 librados: calling monclient init 2024-03-25T19:44:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.491+0000 7f2d5ce0f640 10 monclient: init 2024-03-25T19:44:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.492+0000 7f2d5ce0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:25.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.492+0000 7f2d5ce0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.492+0000 7f2d5ce0f640 10 monclient(hunting): picked mon.noname-b con 0x7f2d580b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.492+0000 7f2d5ce0f640 10 monclient(hunting): picked mon.noname-a con 0x7f2d580b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.492+0000 7f2d5ce0f640 10 monclient(hunting): picked mon.noname-c con 0x7f2d5814be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.492+0000 7f2d5ce0f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d581363f0 con 0x7f2d580b4d60 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.492+0000 7f2d5ce0f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d581374e0 con 0x7f2d580b1730 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.492+0000 7f2d5ce0f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d58138140 con 0x7f2d5814be90 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.492+0000 7f2d5ce0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.492+0000 7f2d5ce0f640 10 monclient(hunting): authenticate will time out at 5057.833242s 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.493+0000 7f2d5659c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2d580b1730 0x7f2d58136dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:60066/0 (socket says 172.21.15.107:60066) 2024-03-25T19:44:25.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.493+0000 7f2d5659c640 1 -- 172.21.15.107:0/2649143818 learned_addr learned my addr 172.21.15.107:0/2649143818 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:25.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.496+0000 7f2d4effd640 10 client.?.objecter ms_handle_connect 0x7f2d580b1730 2024-03-25T19:44:25.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.496+0000 7f2d4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:25.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.504+0000 7f2d4effd640 10 client.?.objecter ms_handle_connect 0x7f2d580b4d60 2024-03-25T19:44:25.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.504+0000 7f2d4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:25.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.506+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149243226 0 0) 0x7f2d581374e0 con 0x7f2d580b1730 2024-03-25T19:44:25.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.506+0000 7f2d4effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:25.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.506+0000 7f2d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:25.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.506+0000 7f2d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:25.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.506+0000 7f2d4effd640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:44:25.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.506+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d38004eb0 con 0x7f2d580b1730 2024-03-25T19:44:25.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.508+0000 7f2d4effd640 10 client.?.objecter ms_handle_connect 0x7f2d5814be90 2024-03-25T19:44:25.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.508+0000 7f2d4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:25.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.510+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480697112 0 0) 0x7f2d581363f0 con 0x7f2d580b4d60 2024-03-25T19:44:25.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.510+0000 7f2d4effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:25.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.510+0000 7f2d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:25.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.510+0000 7f2d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:25.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.510+0000 7f2d4effd640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:44:25.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.510+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d581374e0 con 0x7f2d580b4d60 2024-03-25T19:44:25.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.516+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344854720 0 0) 0x7f2d58138140 con 0x7f2d5814be90 2024-03-25T19:44:25.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.516+0000 7f2d4effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:25.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.516+0000 7f2d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:25.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.516+0000 7f2d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:25.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.516+0000 7f2d4effd640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:44:25.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.516+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d581363f0 con 0x7f2d5814be90 2024-03-25T19:44:25.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.518+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1535315273 0 0) 0x7f2d581374e0 con 0x7f2d580b4d60 2024-03-25T19:44:25.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.518+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d58138140 con 0x7f2d580b4d60 2024-03-25T19:44:25.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.518+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2d400018b0 con 0x7f2d580b4d60 2024-03-25T19:44:25.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.518+0000 7f2d4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:25.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.518+0000 7f2d4effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.518+0000 7f2d4effd640 10 monclient(hunting): dump: 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:25.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.518+0000 7f2d4effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:25.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.522+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 498077562 0 0) 0x7f2d38004eb0 con 0x7f2d580b1730 2024-03-25T19:44:25.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.522+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d581374e0 con 0x7f2d580b1730 2024-03-25T19:44:25.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.522+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2d480019f0 con 0x7f2d580b1730 2024-03-25T19:44:25.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.522+0000 7f2d4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:25.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.522+0000 7f2d4effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:25.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.522+0000 7f2d4effd640 10 monclient(hunting): dump: 2024-03-25T19:44:25.522 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:25.522 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:25.522 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:25.522 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:25.523 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:25.523 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:25.523 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:25.523 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:25.523 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:25.523 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.524+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1116273147 0 0) 0x7f2d58138140 con 0x7f2d580b4d60 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.524+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 >> v1:172.21.15.107:6790/0 conn(0x7f2d5814be90 legacy=0x7f2d58143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.524+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 >> v1:172.21.15.195:6789/0 conn(0x7f2d580b1730 legacy=0x7f2d58136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.524+0000 7f2d4effd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.524+0000 7f2d4effd640 1 monclient: found mon.a 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.524+0000 7f2d4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.524+0000 7f2d4effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.524+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d58139090 con 0x7f2d580b4d60 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.524+0000 7f2d4effd640 10 monclient: _finish_auth 0 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.524+0000 7f2d4effd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.524+0000 7f2d4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.525+0000 7f2d5ce0f640 5 monclient: authenticate success, global_id 4611 2024-03-25T19:44:25.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.525+0000 7f2d5ce0f640 10 monclient: _renew_subs 2024-03-25T19:44:25.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.525+0000 7f2d5ce0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:25.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.525+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2649143818 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d58136030 con 0x7f2d580b4d60 2024-03-25T19:44:25.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.525+0000 7f2d5ce0f640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:25.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.525+0000 7f2d5ce0f640 10 monclient: _renew_subs 2024-03-25T19:44:25.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.525+0000 7f2d5ce0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:25.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.525+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2649143818 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d58130650 con 0x7f2d580b4d60 2024-03-25T19:44:25.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.525+0000 7f2d5ce0f640 1 librados: init done 2024-03-25T19:44:25.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.526+0000 7f2d5ce0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:25.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.526+0000 7f2d5ce0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:25.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.526+0000 7f2d5ce0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:25.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.526+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2649143818 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d580ae6c0 con 0x7f2d580b4d60 2024-03-25T19:44:25.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.527+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d400035f0 con 0x7f2d580b4d60 2024-03-25T19:44:25.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.527+0000 7f2d4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:25.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.527+0000 7f2d5559a640 4 set_mon_vals no callback set 2024-03-25T19:44:25.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.546+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2d400039c0 con 0x7f2d580b4d60 2024-03-25T19:44:25.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.546+0000 7f2d4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.546+0000 7f2d4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.546+0000 7f2d4effd640 10 monclient: dump: 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:25.546 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:25.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.547+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f2d40003ba0 con 0x7f2d580b4d60 2024-03-25T19:44:25.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.547+0000 7f2d5659c640 10 monclient: get_auth_request con 0x7f2d3806df20 auth_method 0 2024-03-25T19:44:25.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.548+0000 7f2d4effd640 10 client.4611.objecter ms_handle_connect 0x7f2d3806df20 2024-03-25T19:44:25.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.552+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f2d40053680 con 0x7f2d580b4d60 2024-03-25T19:44:25.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.552+0000 7f2d4effd640 10 client.4611.objecter ms_dispatch 0x7f2d58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:25.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.552+0000 7f2d4effd640 3 client.4611.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:25.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.552+0000 7f2d4effd640 3 client.4611.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:25.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.552+0000 7f2d4effd640 20 client.4611.objecter dump_active .. 0 homeless 2024-03-25T19:44:25.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.827+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d4004f190 con 0x7f2d580b4d60 2024-03-25T19:44:25.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.827+0000 7f2d4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:25.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:25.827+0000 7f2d4effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:25.920 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:08:56.236 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:25.936 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:08:56.252 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:25.963 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:08:56.279 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:25.979 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:08:56.265 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:25.986 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:08:56.271 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:25.987 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:08:56.303 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:25.999 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:08:56.315 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:26.010 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:08:56.297 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:26.022 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:08:56.338 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:26.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.068+0000 7f2d5ce0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:26.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.068+0000 7f2d5ce0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:26.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.068+0000 7f2d5ce0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:26.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.068+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2649143818 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d58143990 con 0x7f2d580b4d60 2024-03-25T19:44:26.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.127+0000 7f2d4effd640 1 -- 172.21.15.107:0/2649143818 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f2d58143990 con 0x7f2d580b4d60 2024-03-25T19:44:26.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.127+0000 7f2d4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:26.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.127+0000 7f2d4effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:26.128 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:44:26.128 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:26.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.129+0000 7f2d5ce0f640 10 librados: watch_flush enter 2024-03-25T19:44:26.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.129+0000 7f2d5ce0f640 10 librados: watch_flush exit 2024-03-25T19:44:26.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.129+0000 7f2d5ce0f640 20 client.4611.objecter shutdown clearing up homeless session... 2024-03-25T19:44:26.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.129+0000 7f2d5ce0f640 10 client.4611.objecter successfully canceled tick 2024-03-25T19:44:26.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.130+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2649143818 >> v1:172.21.15.195:6800/2984408421 conn(0x7f2d3806df20 legacy=0x7f2d380703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:26.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.130+0000 7f2d5ce0f640 10 monclient: shutdown 2024-03-25T19:44:26.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.130+0000 7f2d5ce0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:26.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.130+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2649143818 >> v1:172.21.15.107:6789/0 conn(0x7f2d580b4d60 legacy=0x7f2d5812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:26.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.130+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2649143818 shutdown_connections 2024-03-25T19:44:26.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.130+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2649143818 >> 172.21.15.107:0/2649143818 conn(0x7f2d580617a0 msgr2=0x7f2d5813c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:26.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.130+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2649143818 shutdown_connections 2024-03-25T19:44:26.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.130+0000 7f2d5ce0f640 1 -- 172.21.15.107:0/2649143818 wait complete. 2024-03-25T19:44:26.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:26.130+0000 7f2d5ce0f640 1 librados: shutdown 2024-03-25T19:44:26.141 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:32.948 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:33.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.066+0000 7f92ebd9d640 10 monclient: get_monmap_and_config 2024-03-25T19:44:33.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.066+0000 7f92ebd9d640 10 monclient: build_initial_monmap 2024-03-25T19:44:33.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.066+0000 7f92ebd9d640 10 monclient: monmap: 2024-03-25T19:44:33.066 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:33.066 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:33.066 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:44:33.067540+0000 2024-03-25T19:44:33.066 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:44:33.067540+0000 2024-03-25T19:44:33.067 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:33.067 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:33.067 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:33.067 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:44:33.067 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:44:33.067 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:33.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.067+0000 7f92ebd9d640 1 Processor -- start 2024-03-25T19:44:33.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.067+0000 7f92ebd9d640 1 -- start start 2024-03-25T19:44:33.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.067+0000 7f92ebd9d640 10 monclient: init 2024-03-25T19:44:33.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.068+0000 7f92ebd9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:33.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.068+0000 7f92ebd9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:33.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.068+0000 7f92ebd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f92e40a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:33.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.068+0000 7f92ebd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f92e413f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:33.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.068+0000 7f92ebd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f92e41430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:33.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.068+0000 7f92ebd9d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e4142520 con 0x7f92e40a98f0 2024-03-25T19:44:33.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.068+0000 7f92ebd9d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e40b28a0 con 0x7f92e41430c0 2024-03-25T19:44:33.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.068+0000 7f92ebd9d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e40b2a30 con 0x7f92e413f710 2024-03-25T19:44:33.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.068+0000 7f92ebd9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:33.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.068+0000 7f92ebd9d640 10 monclient(hunting): authenticate will time out at 5065.409187s 2024-03-25T19:44:33.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.073+0000 7f92e9338640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7f92e413f710 legacy=0x7f92e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T19:44:33.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.073+0000 7f92e9338640 1 -- >> v1:172.21.15.107:6790/0 conn(0x7f92e413f710 legacy=0x7f92e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:44:33.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.073+0000 7f92e9338640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f92e413f710 0x7f92e41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-25T19:44:33.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.078+0000 7f92e9b39640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f92e40a98f0 0x7f92e40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38234/0 (socket says 172.21.15.107:38234) 2024-03-25T19:44:33.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.078+0000 7f92e9b39640 1 -- 172.21.15.107:0/2036808537 learned_addr learned my addr 172.21.15.107:0/2036808537 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:33.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.096+0000 7f92e8b37640 1 -- 172.21.15.107:0/2036808537 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044791736 0 0) 0x7f92e40b28a0 con 0x7f92e41430c0 2024-03-25T19:44:33.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.096+0000 7f92e8b37640 10 monclient(hunting): _negotiate 2024-03-25T19:44:33.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.096+0000 7f92e8b37640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:33.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.096+0000 7f92e8b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:33.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.096+0000 7f92e8b37640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:44:33.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.096+0000 7f92e8b37640 1 -- 172.21.15.107:0/2036808537 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d0003520 con 0x7f92e41430c0 2024-03-25T19:44:33.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.103+0000 7f92e8b37640 1 -- 172.21.15.107:0/2036808537 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794723084 0 0) 0x7f92e4142520 con 0x7f92e40a98f0 2024-03-25T19:44:33.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.103+0000 7f92e8b37640 10 monclient(hunting): _negotiate 2024-03-25T19:44:33.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.103+0000 7f92e8b37640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:33.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.103+0000 7f92e8b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:33.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.103+0000 7f92e8b37640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:44:33.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.103+0000 7f92e8b37640 1 -- 172.21.15.107:0/2036808537 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e40b28a0 con 0x7f92e40a98f0 2024-03-25T19:44:33.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 1 -- 172.21.15.107:0/2036808537 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4274201632 0 0) 0x7f92e40b28a0 con 0x7f92e40a98f0 2024-03-25T19:44:33.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 1 -- 172.21.15.107:0/2036808537 >> v1:172.21.15.107:6790/0 conn(0x7f92e413f710 legacy=0x7f92e41429b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:44:33.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 1 -- 172.21.15.107:0/2036808537 >> v1:172.21.15.195:6789/0 conn(0x7f92e41430c0 legacy=0x7f92e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:33.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 10 monclient: _finish_hunting 0 2024-03-25T19:44:33.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 1 monclient: found mon.noname-a 2024-03-25T19:44:33.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:33.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:44:33.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 1 -- 172.21.15.107:0/2036808537 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e40b44c0 con 0x7f92e40a98f0 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 10 monclient: _finish_auth 0 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 10 monclient: _check_auth_tickets 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 1 -- 172.21.15.107:0/2036808537 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f92e0001670 con 0x7f92e40a98f0 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92e8b37640 10 monclient: dump: 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:33.118 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:33.119 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:33.119 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:33.119 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:33.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92ebd9d640 5 monclient: authenticate success, global_id 4614 2024-03-25T19:44:33.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.117+0000 7f92ebd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:33.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.122+0000 7f92e8b37640 1 -- 172.21.15.107:0/2036808537 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92e00017d0 con 0x7f92e40a98f0 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.122+0000 7f92e8b37640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.122+0000 7f92e8b37640 1 -- 172.21.15.107:0/2036808537 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f92e0001c20 con 0x7f92e40a98f0 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.122+0000 7f92e8b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.122+0000 7f92e8b37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.122+0000 7f92e8b37640 10 monclient: dump: 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:33.122 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:33.123 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:33.123 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:33.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.122+0000 7f92ebd9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:33.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.122+0000 7f92ebd9d640 4 set_mon_vals no callback set 2024-03-25T19:44:33.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.123+0000 7f92ebd9d640 10 monclient: shutdown 2024-03-25T19:44:33.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.123+0000 7f92ebd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:33.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.123+0000 7f92ebd9d640 1 -- 172.21.15.107:0/2036808537 >> v1:172.21.15.107:6789/0 conn(0x7f92e40a98f0 legacy=0x7f92e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:33.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.123+0000 7f92ebd9d640 1 -- 172.21.15.107:0/2036808537 shutdown_connections 2024-03-25T19:44:33.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.123+0000 7f92ebd9d640 1 -- 172.21.15.107:0/2036808537 >> 172.21.15.107:0/2036808537 conn(0x7f92e40b7ea0 msgr2=0x7f92e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:33.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.123+0000 7f92ebd9d640 1 -- 172.21.15.107:0/2036808537 shutdown_connections 2024-03-25T19:44:33.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.123+0000 7f92ebd9d640 1 -- 172.21.15.107:0/2036808537 wait complete. 2024-03-25T19:44:33.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.124+0000 7f92ebd9d640 10 monclient: build_initial_monmap 2024-03-25T19:44:33.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.124+0000 7f92ebd9d640 10 monclient: monmap: 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.124+0000 7f92ebd9d640 1 librados: starting msgr at 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.124+0000 7f92ebd9d640 1 librados: starting objecter 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.124+0000 7f92ebd9d640 1 Processor -- start 2024-03-25T19:44:33.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.124+0000 7f92ebd9d640 1 -- start start 2024-03-25T19:44:33.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.124+0000 7f92ebd9d640 1 librados: setting wanted keys 2024-03-25T19:44:33.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.124+0000 7f92ebd9d640 1 librados: calling monclient init 2024-03-25T19:44:33.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.124+0000 7f92ebd9d640 10 monclient: init 2024-03-25T19:44:33.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.125+0000 7f92ebd9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:33.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.125+0000 7f92ebd9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:33.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.125+0000 7f92ebd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f92e40a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:33.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.125+0000 7f92ebd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f92e413f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:33.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.125+0000 7f92ebd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f92e41430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:33.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.125+0000 7f92ebd9d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e414e720 con 0x7f92e40a98f0 2024-03-25T19:44:33.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.125+0000 7f92ebd9d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e414bcb0 con 0x7f92e41430c0 2024-03-25T19:44:33.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.125+0000 7f92ebd9d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e414be40 con 0x7f92e413f710 2024-03-25T19:44:33.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.125+0000 7f92ebd9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:33.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.125+0000 7f92ebd9d640 10 monclient(hunting): authenticate will time out at 5065.466576s 2024-03-25T19:44:33.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.136+0000 7f92ea33a640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f92e41430c0 0x7f92e41e7020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:60298/0 (socket says 172.21.15.107:60298) 2024-03-25T19:44:33.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.136+0000 7f92ea33a640 1 -- 172.21.15.107:0/678276963 learned_addr learned my addr 172.21.15.107:0/678276963 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:33.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.137+0000 7f92da7fc640 10 client.?.objecter ms_handle_connect 0x7f92e41430c0 2024-03-25T19:44:33.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.137+0000 7f92da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:33.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.140+0000 7f92da7fc640 10 client.?.objecter ms_handle_connect 0x7f92e40a98f0 2024-03-25T19:44:33.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.140+0000 7f92da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:33.140 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.140+0000 7f92da7fc640 10 client.?.objecter ms_handle_connect 0x7f92e413f710 2024-03-25T19:44:33.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.140+0000 7f92da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:33.143 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.143+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477593072 0 0) 0x7f92e414bcb0 con 0x7f92e41430c0 2024-03-25T19:44:33.143 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.143+0000 7f92da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:33.143 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.143+0000 7f92da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:33.143 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.143+0000 7f92da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:33.143 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.143+0000 7f92da7fc640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:44:33.143 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.143+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c8004eb0 con 0x7f92e41430c0 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.150+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049667124 0 0) 0x7f92e414be40 con 0x7f92e413f710 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.150+0000 7f92da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.150+0000 7f92da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.150+0000 7f92da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.150+0000 7f92da7fc640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:44:33.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.150+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e414bcb0 con 0x7f92e413f710 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.155+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231151201 0 0) 0x7f92e414e720 con 0x7f92e40a98f0 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.155+0000 7f92da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.155+0000 7f92da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.155+0000 7f92da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:33.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.155+0000 7f92da7fc640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:44:33.156 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.155+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e414be40 con 0x7f92e40a98f0 2024-03-25T19:44:33.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.167+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2522794288 0 0) 0x7f92c8004eb0 con 0x7f92e41430c0 2024-03-25T19:44:33.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.167+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92e414e720 con 0x7f92e41430c0 2024-03-25T19:44:33.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.167+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f92dc001c70 con 0x7f92e41430c0 2024-03-25T19:44:33.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.167+0000 7f92da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:33.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.167+0000 7f92da7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.167+0000 7f92da7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:33.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.167+0000 7f92da7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.170+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1176830971 0 0) 0x7f92e414bcb0 con 0x7f92e413f710 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.170+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c8004eb0 con 0x7f92e413f710 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.170+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f92d4001850 con 0x7f92e413f710 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.170+0000 7f92da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.170+0000 7f92da7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.170+0000 7f92da7fc640 10 monclient(hunting): dump: 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:33.170 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:33.171 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:33.171 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:33.171 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:33.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1880333494 0 0) 0x7f92e414e720 con 0x7f92e41430c0 2024-03-25T19:44:33.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 >> v1:172.21.15.107:6790/0 conn(0x7f92e413f710 legacy=0x7f92e41db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:33.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 >> v1:172.21.15.107:6789/0 conn(0x7f92e40a98f0 legacy=0x7f92e40714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:33.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92da7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:33.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92da7fc640 1 monclient: found mon.b 2024-03-25T19:44:33.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92da7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e414bfd0 con 0x7f92e41430c0 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92da7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92da7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92ebd9d640 5 monclient: authenticate success, global_id 4606 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92ebd9d640 10 monclient: _renew_subs 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92ebd9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.177+0000 7f92ebd9d640 1 -- 172.21.15.107:0/678276963 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e406dea0 con 0x7f92e41430c0 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.178+0000 7f92ebd9d640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.178+0000 7f92ebd9d640 10 monclient: _renew_subs 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.178+0000 7f92ebd9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.178+0000 7f92ebd9d640 1 -- 172.21.15.107:0/678276963 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e406e940 con 0x7f92e41430c0 2024-03-25T19:44:33.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.178+0000 7f92ebd9d640 1 librados: init done 2024-03-25T19:44:33.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.179+0000 7f92ebd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:33.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.179+0000 7f92ebd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:33.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.179+0000 7f92ebd9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:44:33.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.179+0000 7f92ebd9d640 1 -- 172.21.15.107:0/678276963 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e40627b0 con 0x7f92e41430c0 2024-03-25T19:44:33.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.183+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92dc003990 con 0x7f92e41430c0 2024-03-25T19:44:33.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.183+0000 7f92da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:33.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.183+0000 7f92dbfff640 4 set_mon_vals no callback set 2024-03-25T19:44:33.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.457+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f92dc003d20 con 0x7f92e41430c0 2024-03-25T19:44:33.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.457+0000 7f92da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:33.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.457+0000 7f92da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:33.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.457+0000 7f92da7fc640 10 monclient: dump: 2024-03-25T19:44:33.457 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:33.458 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:33.458 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:33.458 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:33.458 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:33.458 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:33.458 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:33.458 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:33.458 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:33.458 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:33.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.458+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f92dc003f00 con 0x7f92e41430c0 2024-03-25T19:44:33.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.459+0000 7f92e9b39640 10 monclient: get_auth_request con 0x7f92c806df70 auth_method 0 2024-03-25T19:44:33.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.459+0000 7f92da7fc640 10 client.4606.objecter ms_handle_connect 0x7f92c806df70 2024-03-25T19:44:33.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.462+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f92dc054b60 con 0x7f92e41430c0 2024-03-25T19:44:33.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.462+0000 7f92da7fc640 10 client.4606.objecter ms_dispatch 0x7f92e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:33.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.462+0000 7f92da7fc640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:33.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.462+0000 7f92da7fc640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:33.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.462+0000 7f92da7fc640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T19:44:33.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.464+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92dc050670 con 0x7f92e41430c0 2024-03-25T19:44:33.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.464+0000 7f92da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:33.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.464+0000 7f92da7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:33.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.694+0000 7f92ebd9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:33.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.694+0000 7f92ebd9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:33.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.694+0000 7f92ebd9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:44:33.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.694+0000 7f92ebd9d640 1 -- 172.21.15.107:0/678276963 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92e414ed60 con 0x7f92e41430c0 2024-03-25T19:44:33.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.725+0000 7f92da7fc640 1 -- 172.21.15.107:0/678276963 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f92dc0525a0 con 0x7f92e41430c0 2024-03-25T19:44:33.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.725+0000 7f92da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:33.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.725+0000 7f92da7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:33.726 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:44:33.726 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:33.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 10 librados: watch_flush enter 2024-03-25T19:44:33.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 10 librados: watch_flush exit 2024-03-25T19:44:33.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T19:44:33.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 10 client.4606.objecter successfully canceled tick 2024-03-25T19:44:33.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 1 -- 172.21.15.107:0/678276963 >> v1:172.21.15.195:6800/2984408421 conn(0x7f92c806df70 legacy=0x7f92c8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:33.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 10 monclient: shutdown 2024-03-25T19:44:33.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:33.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 1 -- 172.21.15.107:0/678276963 >> v1:172.21.15.195:6789/0 conn(0x7f92e41430c0 legacy=0x7f92e41e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:33.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 1 -- 172.21.15.107:0/678276963 shutdown_connections 2024-03-25T19:44:33.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 1 -- 172.21.15.107:0/678276963 >> 172.21.15.107:0/678276963 conn(0x7f92e40b7ea0 msgr2=0x7f92e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:33.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 1 -- 172.21.15.107:0/678276963 shutdown_connections 2024-03-25T19:44:33.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 1 -- 172.21.15.107:0/678276963 wait complete. 2024-03-25T19:44:33.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:33.728+0000 7f92ebd9d640 1 librados: shutdown 2024-03-25T19:44:33.739 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:40.546 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:40.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.664+0000 7f6d4ae40640 10 monclient: get_monmap_and_config 2024-03-25T19:44:40.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.664+0000 7f6d4ae40640 10 monclient: build_initial_monmap 2024-03-25T19:44:40.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.664+0000 7f6d4ae40640 10 monclient: monmap: 2024-03-25T19:44:40.664 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:40.664 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:40.664 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:44:40.665126+0000 2024-03-25T19:44:40.664 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:44:40.665126+0000 2024-03-25T19:44:40.664 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:40.664 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:40.664 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:40.665 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:44:40.665 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:44:40.665 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:40.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 1 Processor -- start 2024-03-25T19:44:40.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 1 -- start start 2024-03-25T19:44:40.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 10 monclient: init 2024-03-25T19:44:40.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:40.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:40.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 10 monclient(hunting): picked mon.noname-b con 0x7f6d440a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:40.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 10 monclient(hunting): picked mon.noname-c con 0x7f6d4413f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:40.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 10 monclient(hunting): picked mon.noname-a con 0x7f6d441430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:40.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d44142520 con 0x7f6d441430c0 2024-03-25T19:44:40.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d440b28a0 con 0x7f6d4413f710 2024-03-25T19:44:40.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d440b2a30 con 0x7f6d440a98f0 2024-03-25T19:44:40.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:40.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.665+0000 7f6d4ae40640 10 monclient(hunting): authenticate will time out at 5073.006573s 2024-03-25T19:44:40.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.671+0000 7f6d48bdc640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6d440a98f0 0x7f6d440a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:57794/0 (socket says 172.21.15.107:57794) 2024-03-25T19:44:40.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.671+0000 7f6d48bdc640 1 -- 172.21.15.107:0/2773204867 learned_addr learned my addr 172.21.15.107:0/2773204867 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:40.690 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.691+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731598499 0 0) 0x7f6d440b2a30 con 0x7f6d440a98f0 2024-03-25T19:44:40.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.691+0000 7f6d3b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:40.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.691+0000 7f6d3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:40.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.691+0000 7f6d3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:40.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.691+0000 7f6d3b7fe640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:44:40.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.691+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d2c003520 con 0x7f6d440a98f0 2024-03-25T19:44:40.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.695+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399591590 0 0) 0x7f6d44142520 con 0x7f6d441430c0 2024-03-25T19:44:40.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.695+0000 7f6d3b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:40.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.695+0000 7f6d3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:40.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.695+0000 7f6d3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:40.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.695+0000 7f6d3b7fe640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:44:40.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.695+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d440b2a30 con 0x7f6d441430c0 2024-03-25T19:44:40.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.698+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659457975 0 0) 0x7f6d440b28a0 con 0x7f6d4413f710 2024-03-25T19:44:40.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.698+0000 7f6d3b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:40.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.698+0000 7f6d3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:40.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.698+0000 7f6d3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:40.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.698+0000 7f6d3b7fe640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:44:40.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.698+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d44142520 con 0x7f6d4413f710 2024-03-25T19:44:40.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3688665657 0 0) 0x7f6d44142520 con 0x7f6d4413f710 2024-03-25T19:44:40.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 >> v1:172.21.15.107:6790/0 conn(0x7f6d440a98f0 legacy=0x7f6d440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 >> v1:172.21.15.107:6789/0 conn(0x7f6d441430c0 legacy=0x7f6d4414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d440b44c0 con 0x7f6d4413f710 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6d340015b0 con 0x7f6d4413f710 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d4ae40640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d4ae40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:40.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.705+0000 7f6d3b7fe640 10 monclient: dump: 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:40.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.706+0000 7f6d4ae40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.717+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d34001710 con 0x7f6d4413f710 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.717+0000 7f6d3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.717+0000 7f6d3b7fe640 1 -- 172.21.15.107:0/2773204867 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6d34001b60 con 0x7f6d4413f710 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.717+0000 7f6d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.717+0000 7f6d3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.717+0000 7f6d3b7fe640 10 monclient: dump: 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:40.717 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:40.718 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:40.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.717+0000 7f6d4ae40640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:40.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.717+0000 7f6d4ae40640 4 set_mon_vals no callback set 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.717+0000 7f6d4ae40640 10 monclient: shutdown 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.717+0000 7f6d4ae40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.717+0000 7f6d4ae40640 1 -- 172.21.15.107:0/2773204867 >> v1:172.21.15.195:6789/0 conn(0x7f6d4413f710 legacy=0x7f6d441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.718+0000 7f6d4ae40640 1 -- 172.21.15.107:0/2773204867 shutdown_connections 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.718+0000 7f6d4ae40640 1 -- 172.21.15.107:0/2773204867 >> 172.21.15.107:0/2773204867 conn(0x7f6d440b7ea0 msgr2=0x7f6d440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.718+0000 7f6d4ae40640 1 -- 172.21.15.107:0/2773204867 shutdown_connections 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.718+0000 7f6d4ae40640 1 -- 172.21.15.107:0/2773204867 wait complete. 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.718+0000 7f6d4ae40640 10 monclient: build_initial_monmap 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.718+0000 7f6d4ae40640 10 monclient: monmap: 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:40.720 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.719+0000 7f6d4ae40640 1 librados: starting msgr at 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.719+0000 7f6d4ae40640 1 librados: starting objecter 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.719+0000 7f6d4ae40640 1 Processor -- start 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.719+0000 7f6d4ae40640 1 -- start start 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.719+0000 7f6d4ae40640 1 librados: setting wanted keys 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.719+0000 7f6d4ae40640 1 librados: calling monclient init 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.719+0000 7f6d4ae40640 10 monclient: init 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.720+0000 7f6d4ae40640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.720+0000 7f6d4ae40640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.720+0000 7f6d4ae40640 10 monclient(hunting): picked mon.noname-b con 0x7f6d440a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:40.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.720+0000 7f6d4ae40640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4413f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:40.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.720+0000 7f6d4ae40640 10 monclient(hunting): picked mon.noname-c con 0x7f6d441430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:40.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.720+0000 7f6d4ae40640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d441db7f0 con 0x7f6d4413f710 2024-03-25T19:44:40.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.720+0000 7f6d4ae40640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d440abd70 con 0x7f6d440a98f0 2024-03-25T19:44:40.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.720+0000 7f6d4ae40640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d44138ca0 con 0x7f6d441430c0 2024-03-25T19:44:40.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.720+0000 7f6d4ae40640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:40.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.720+0000 7f6d4ae40640 10 monclient(hunting): authenticate will time out at 5073.061012s 2024-03-25T19:44:40.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.725+0000 7f6d493dd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6d441430c0 0x7f6d44138590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:57840/0 (socket says 172.21.15.107:57840) 2024-03-25T19:44:40.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.726+0000 7f6d493dd640 1 -- 172.21.15.107:0/1092505111 learned_addr learned my addr 172.21.15.107:0/1092505111 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:40.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.726+0000 7f6d397fa640 10 client.?.objecter ms_handle_connect 0x7f6d441430c0 2024-03-25T19:44:40.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.726+0000 7f6d397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:40.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.733+0000 7f6d397fa640 10 client.?.objecter ms_handle_connect 0x7f6d440a98f0 2024-03-25T19:44:40.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.733+0000 7f6d397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:40.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.737+0000 7f6d397fa640 10 client.?.objecter ms_handle_connect 0x7f6d4413f710 2024-03-25T19:44:40.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.737+0000 7f6d397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:40.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.740+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189720446 0 0) 0x7f6d44138ca0 con 0x7f6d441430c0 2024-03-25T19:44:40.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.740+0000 7f6d397fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:40.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.740+0000 7f6d397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:40.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.740+0000 7f6d397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:40.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.740+0000 7f6d397fa640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:44:40.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.740+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d30004eb0 con 0x7f6d441430c0 2024-03-25T19:44:40.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.747+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285974305 0 0) 0x7f6d441db7f0 con 0x7f6d4413f710 2024-03-25T19:44:40.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.747+0000 7f6d397fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:40.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.747+0000 7f6d397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:40.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.747+0000 7f6d397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.747+0000 7f6d397fa640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:44:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.747+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d44138ca0 con 0x7f6d4413f710 2024-03-25T19:44:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.748+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220349103 0 0) 0x7f6d440abd70 con 0x7f6d440a98f0 2024-03-25T19:44:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.748+0000 7f6d397fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.748+0000 7f6d397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.748+0000 7f6d397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.748+0000 7f6d397fa640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:44:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.748+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d441db7f0 con 0x7f6d440a98f0 2024-03-25T19:44:40.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.759+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3619777162 0 0) 0x7f6d44138ca0 con 0x7f6d4413f710 2024-03-25T19:44:40.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d440abd70 con 0x7f6d4413f710 2024-03-25T19:44:40.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6d34001a40 con 0x7f6d4413f710 2024-03-25T19:44:40.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:40.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:40.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 10 monclient(hunting): dump: 2024-03-25T19:44:40.760 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:40.760 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:40.760 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:40.760 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2792629077 0 0) 0x7f6d441db7f0 con 0x7f6d440a98f0 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d44138ca0 con 0x7f6d440a98f0 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6d40001c50 con 0x7f6d440a98f0 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.760+0000 7f6d397fa640 10 monclient(hunting): dump: 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:40.761 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:40.762 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:40.762 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:40.762 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:40.762 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:40.762 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:40.762 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:40.762 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:40.762 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:40.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.767+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3699190063 0 0) 0x7f6d30004eb0 con 0x7f6d441430c0 2024-03-25T19:44:40.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.768+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d441db7f0 con 0x7f6d441430c0 2024-03-25T19:44:40.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.768+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6d3c001910 con 0x7f6d441430c0 2024-03-25T19:44:40.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.768+0000 7f6d397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.768+0000 7f6d397fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.768+0000 7f6d397fa640 10 monclient(hunting): dump: 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:40.768 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 275065274 0 0) 0x7f6d441db7f0 con 0x7f6d441430c0 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 >> v1:172.21.15.195:6789/0 conn(0x7f6d440a98f0 legacy=0x7f6d44137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 >> v1:172.21.15.107:6789/0 conn(0x7f6d4413f710 legacy=0x7f6d441db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d397fa640 10 monclient: _finish_hunting 0 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d397fa640 1 monclient: found mon.c 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d397fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d44138e30 con 0x7f6d441430c0 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d397fa640 10 monclient: _finish_auth 0 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d397fa640 10 monclient: _check_auth_tickets 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d4ae40640 5 monclient: authenticate success, global_id 4625 2024-03-25T19:44:40.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d4ae40640 10 monclient: _renew_subs 2024-03-25T19:44:40.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d4ae40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:44:40.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d4ae40640 1 -- 172.21.15.107:0/1092505111 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d441e2c30 con 0x7f6d441430c0 2024-03-25T19:44:40.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d4ae40640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:40.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d4ae40640 10 monclient: _renew_subs 2024-03-25T19:44:40.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d4ae40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:44:40.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d4ae40640 1 -- 172.21.15.107:0/1092505111 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d441e2fa0 con 0x7f6d441430c0 2024-03-25T19:44:40.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.775+0000 7f6d4ae40640 1 librados: init done 2024-03-25T19:44:40.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.776+0000 7f6d4ae40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:40.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.776+0000 7f6d4ae40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:40.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.776+0000 7f6d4ae40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:44:40.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:40.776+0000 7f6d4ae40640 1 -- 172.21.15.107:0/1092505111 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d440b3ef0 con 0x7f6d441430c0 2024-03-25T19:44:41.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.089+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d3c0035f0 con 0x7f6d441430c0 2024-03-25T19:44:41.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.089+0000 7f6d397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:41.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.089+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f6d3c003900 con 0x7f6d441430c0 2024-03-25T19:44:41.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.089+0000 7f6d397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:41.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.089+0000 7f6d397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:41.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.089+0000 7f6d397fa640 10 monclient: dump: 2024-03-25T19:44:41.089 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:41.089 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:41.090 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:41.090 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:41.090 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:41.090 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:41.090 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:41.090 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:41.090 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:41.090 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:41.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.089+0000 7f6d3affd640 4 set_mon_vals no callback set 2024-03-25T19:44:41.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.090+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f6d3c003b20 con 0x7f6d441430c0 2024-03-25T19:44:41.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.091+0000 7f6d48bdc640 10 monclient: get_auth_request con 0x7f6d3006e110 auth_method 0 2024-03-25T19:44:41.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.091+0000 7f6d397fa640 10 client.4625.objecter ms_handle_connect 0x7f6d3006e110 2024-03-25T19:44:41.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.100+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f6d3c0546e0 con 0x7f6d441430c0 2024-03-25T19:44:41.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.100+0000 7f6d397fa640 10 client.4625.objecter ms_dispatch 0x7f6d44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:41.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.100+0000 7f6d397fa640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:41.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.100+0000 7f6d397fa640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:41.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.100+0000 7f6d397fa640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-25T19:44:41.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.100+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d3c0501f0 con 0x7f6d441430c0 2024-03-25T19:44:41.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.100+0000 7f6d397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:41.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.100+0000 7f6d397fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:41.341 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.341+0000 7f6d4ae40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:41.341 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.341+0000 7f6d4ae40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:41.341 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.341+0000 7f6d4ae40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:44:41.341 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.341+0000 7f6d4ae40640 1 -- 172.21.15.107:0/1092505111 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d44137a50 con 0x7f6d441430c0 2024-03-25T19:44:41.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.390+0000 7f6d397fa640 1 -- 172.21.15.107:0/1092505111 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f6d3c052120 con 0x7f6d441430c0 2024-03-25T19:44:41.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.390+0000 7f6d397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:41.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.390+0000 7f6d397fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:41.390 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:44:41.390 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:41.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 10 librados: watch_flush enter 2024-03-25T19:44:41.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 10 librados: watch_flush exit 2024-03-25T19:44:41.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-25T19:44:41.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 10 client.4625.objecter successfully canceled tick 2024-03-25T19:44:41.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 1 -- 172.21.15.107:0/1092505111 >> v1:172.21.15.195:6800/2984408421 conn(0x7f6d3006e110 legacy=0x7f6d300705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:41.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 10 monclient: shutdown 2024-03-25T19:44:41.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:41.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 1 -- 172.21.15.107:0/1092505111 >> v1:172.21.15.107:6790/0 conn(0x7f6d441430c0 legacy=0x7f6d44138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:41.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 1 -- 172.21.15.107:0/1092505111 shutdown_connections 2024-03-25T19:44:41.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 1 -- 172.21.15.107:0/1092505111 >> 172.21.15.107:0/1092505111 conn(0x7f6d440b7ea0 msgr2=0x7f6d44074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:41.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 1 -- 172.21.15.107:0/1092505111 shutdown_connections 2024-03-25T19:44:41.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 1 -- 172.21.15.107:0/1092505111 wait complete. 2024-03-25T19:44:41.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:41.392+0000 7f6d4ae40640 1 librados: shutdown 2024-03-25T19:44:41.403 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:46.227 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:09:16.542 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:46.280 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:09:16.596 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:46.319 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:09:16.635 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:46.326 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:09:16.612 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:46.340 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:09:16.627 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:46.361 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:09:16.677 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:46.364 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:09:16.681 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:46.373 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:09:16.660 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:46.391 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:09:16.707 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:48.210 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:48.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.326+0000 7fe7b9d54640 10 monclient: get_monmap_and_config 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.326+0000 7fe7b9d54640 10 monclient: build_initial_monmap 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.326+0000 7fe7b9d54640 10 monclient: monmap: 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:44:48.327238+0000 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:44:48.327238+0000 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:44:48.326 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 1 Processor -- start 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 1 -- start start 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 10 monclient: init 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 10 monclient(hunting): picked mon.noname-c con 0x7fe7b40b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 10 monclient(hunting): picked mon.noname-b con 0x7fe7b40b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 10 monclient(hunting): picked mon.noname-a con 0x7fe7b414be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b40af940 con 0x7fe7b414be90 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b40b0a70 con 0x7fe7b40b4d60 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b40b0d50 con 0x7fe7b40b1730 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.327+0000 7fe7b9d54640 10 monclient(hunting): authenticate will time out at 5080.668380s 2024-03-25T19:44:48.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.328+0000 7fe7b3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe7b414be90 0x7fe7b414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60070/0 (socket says 172.21.15.107:60070) 2024-03-25T19:44:48.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.328+0000 7fe7b3fff640 1 -- 172.21.15.107:0/3901157888 learned_addr learned my addr 172.21.15.107:0/3901157888 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:48.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.406+0000 7fe7b27fc640 1 -- 172.21.15.107:0/3901157888 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400937128 0 0) 0x7fe7b40b0d50 con 0x7fe7b40b1730 2024-03-25T19:44:48.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.406+0000 7fe7b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:48.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.406+0000 7fe7b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:48.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.406+0000 7fe7b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:48.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.406+0000 7fe7b27fc640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:44:48.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.406+0000 7fe7b27fc640 1 -- 172.21.15.107:0/3901157888 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe79c003520 con 0x7fe7b40b1730 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b27fc640 1 -- 172.21.15.107:0/3901157888 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2798666051 0 0) 0x7fe79c003520 con 0x7fe7b40b1730 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b27fc640 1 -- 172.21.15.107:0/3901157888 >> v1:172.21.15.195:6789/0 conn(0x7fe7b40b4d60 legacy=0x7fe7b413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b27fc640 1 -- 172.21.15.107:0/3901157888 >> v1:172.21.15.107:6789/0 conn(0x7fe7b414be90 legacy=0x7fe7b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b27fc640 1 monclient: found mon.noname-b 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b27fc640 1 -- 172.21.15.107:0/3901157888 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b40b1030 con 0x7fe7b40b1730 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b27fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:48.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b9d54640 5 monclient: authenticate success, global_id 4628 2024-03-25T19:44:48.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.422+0000 7fe7b9d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:48.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.423+0000 7fe7b27fc640 1 -- 172.21.15.107:0/3901157888 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe7a8001690 con 0x7fe7b40b1730 2024-03-25T19:44:48.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.423+0000 7fe7b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:48.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.423+0000 7fe7b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:48.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.423+0000 7fe7b27fc640 10 monclient: dump: 2024-03-25T19:44:48.423 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:48.423 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:48.424 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.424 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.424 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:48.424 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:48.424 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:48.424 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:48.424 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:48.424 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:48.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.423+0000 7fe7b9d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:48.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.431+0000 7fe7b27fc640 1 -- 172.21.15.107:0/3901157888 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7a8001920 con 0x7fe7b40b1730 2024-03-25T19:44:48.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.431+0000 7fe7b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:48.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.431+0000 7fe7b27fc640 1 -- 172.21.15.107:0/3901157888 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe7a8001c30 con 0x7fe7b40b1730 2024-03-25T19:44:48.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.431+0000 7fe7b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:48.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.431+0000 7fe7b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:48.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.431+0000 7fe7b27fc640 10 monclient: dump: 2024-03-25T19:44:48.431 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:48.431 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:48.431 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.431 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.432 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:48.432 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:48.432 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:48.432 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:48.432 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:48.432 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:48.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.431+0000 7fe7b9d54640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:48.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.431+0000 7fe7b9d54640 4 set_mon_vals no callback set 2024-03-25T19:44:48.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.432+0000 7fe7b9d54640 10 monclient: shutdown 2024-03-25T19:44:48.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.432+0000 7fe7b9d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:48.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.432+0000 7fe7b9d54640 1 -- 172.21.15.107:0/3901157888 >> v1:172.21.15.107:6790/0 conn(0x7fe7b40b1730 legacy=0x7fe7b40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.432+0000 7fe7b9d54640 1 -- 172.21.15.107:0/3901157888 shutdown_connections 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.432+0000 7fe7b9d54640 1 -- 172.21.15.107:0/3901157888 >> 172.21.15.107:0/3901157888 conn(0x7fe7b40617a0 msgr2=0x7fe7b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.432+0000 7fe7b9d54640 1 -- 172.21.15.107:0/3901157888 shutdown_connections 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.432+0000 7fe7b9d54640 1 -- 172.21.15.107:0/3901157888 wait complete. 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.433+0000 7fe7b9d54640 10 monclient: build_initial_monmap 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.433+0000 7fe7b9d54640 10 monclient: monmap: 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:48.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.433+0000 7fe7b9d54640 1 librados: starting msgr at 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.433+0000 7fe7b9d54640 1 librados: starting objecter 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.433+0000 7fe7b9d54640 1 Processor -- start 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.433+0000 7fe7b9d54640 1 -- start start 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.433+0000 7fe7b9d54640 1 librados: setting wanted keys 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.433+0000 7fe7b9d54640 1 librados: calling monclient init 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.433+0000 7fe7b9d54640 10 monclient: init 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.434+0000 7fe7b9d54640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.434+0000 7fe7b9d54640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.434+0000 7fe7b9d54640 10 monclient(hunting): picked mon.noname-c con 0x7fe7b40b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.434+0000 7fe7b9d54640 10 monclient(hunting): picked mon.noname-a con 0x7fe7b40b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.434+0000 7fe7b9d54640 10 monclient(hunting): picked mon.noname-b con 0x7fe7b414be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.434+0000 7fe7b9d54640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b41423c0 con 0x7fe7b40b4d60 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.434+0000 7fe7b9d54640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b41426a0 con 0x7fe7b414be90 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.434+0000 7fe7b9d54640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b4138180 con 0x7fe7b40b1730 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.434+0000 7fe7b9d54640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:48.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.434+0000 7fe7b9d54640 10 monclient(hunting): authenticate will time out at 5080.774996s 2024-03-25T19:44:48.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.453+0000 7fe7b37fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fe7b40b1730 0x7fe7b4142e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34144/0 (socket says 172.21.15.107:34144) 2024-03-25T19:44:48.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.453+0000 7fe7b37fe640 1 -- 172.21.15.107:0/1218447246 learned_addr learned my addr 172.21.15.107:0/1218447246 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:48.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.459+0000 7fe79bfff640 10 client.?.objecter ms_handle_connect 0x7fe7b40b1730 2024-03-25T19:44:48.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.459+0000 7fe79bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:48.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.464+0000 7fe79bfff640 10 client.?.objecter ms_handle_connect 0x7fe7b414be90 2024-03-25T19:44:48.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.464+0000 7fe79bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.469+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016123217 0 0) 0x7fe7b4138180 con 0x7fe7b40b1730 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.469+0000 7fe79bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.469+0000 7fe79bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:48.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.469+0000 7fe79bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:48.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.469+0000 7fe79bfff640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:44:48.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.469+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a0004eb0 con 0x7fe7b40b1730 2024-03-25T19:44:48.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.470+0000 7fe79bfff640 10 client.?.objecter ms_handle_connect 0x7fe7b40b4d60 2024-03-25T19:44:48.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.470+0000 7fe79bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:48.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.474+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597953834 0 0) 0x7fe7b41426a0 con 0x7fe7b414be90 2024-03-25T19:44:48.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.474+0000 7fe79bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:48.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.474+0000 7fe79bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:48.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.474+0000 7fe79bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:48.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.474+0000 7fe79bfff640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:44:48.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.474+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b4138180 con 0x7fe7b414be90 2024-03-25T19:44:48.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.483+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820967326 0 0) 0x7fe7b41423c0 con 0x7fe7b40b4d60 2024-03-25T19:44:48.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.483+0000 7fe79bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:48.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.483+0000 7fe79bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:48.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.483+0000 7fe79bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:48.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.483+0000 7fe79bfff640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:44:48.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.483+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b41426a0 con 0x7fe7b40b4d60 2024-03-25T19:44:48.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.490+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3159666510 0 0) 0x7fe7b41426a0 con 0x7fe7b40b4d60 2024-03-25T19:44:48.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.490+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7b41423c0 con 0x7fe7b40b4d60 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.490+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2735281626 0 0) 0x7fe7b4138180 con 0x7fe7b414be90 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.490+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7b41426a0 con 0x7fe7b414be90 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.490+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe7ac001a80 con 0x7fe7b414be90 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.490+0000 7fe79bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.490+0000 7fe79bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.490+0000 7fe79bfff640 10 monclient(hunting): dump: 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:48.491 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:48.492 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:48.492 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:48.492 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:48.492 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:48.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.491+0000 7fe79bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:48.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.492+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe7a8001650 con 0x7fe7b40b4d60 2024-03-25T19:44:48.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.492+0000 7fe79bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:48.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.492+0000 7fe79bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:48.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.492+0000 7fe79bfff640 10 monclient(hunting): dump: 2024-03-25T19:44:48.493 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:48.493 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:48.493 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.493 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.493 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:48.493 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:48.493 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:48.493 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:48.493 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:48.493 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.495+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2885153314 0 0) 0x7fe7a0004eb0 con 0x7fe7b40b1730 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.495+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7b4138180 con 0x7fe7b40b1730 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.495+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe7a40019d0 con 0x7fe7b40b1730 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.495+0000 7fe79bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.495+0000 7fe79bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.495+0000 7fe79bfff640 10 monclient(hunting): dump: 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:48.495 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:48.496 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:48.496 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:48.496 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:48.496 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 318566352 0 0) 0x7fe7b41423c0 con 0x7fe7b40b4d60 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 >> v1:172.21.15.107:6790/0 conn(0x7fe7b40b1730 legacy=0x7fe7b4142e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 >> v1:172.21.15.195:6789/0 conn(0x7fe7b414be90 legacy=0x7fe7b4140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe79bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe79bfff640 1 monclient: found mon.a 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe79bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe79bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b4138310 con 0x7fe7b40b4d60 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe79bfff640 10 monclient: _finish_auth 0 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe79bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe79bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.502+0000 7fe7b9d54640 5 monclient: authenticate success, global_id 4626 2024-03-25T19:44:48.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.503+0000 7fe7b9d54640 10 monclient: _renew_subs 2024-03-25T19:44:48.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.503+0000 7fe7b9d54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:48.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.503+0000 7fe7b9d54640 1 -- 172.21.15.107:0/1218447246 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b41e3960 con 0x7fe7b40b4d60 2024-03-25T19:44:48.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.503+0000 7fe7b9d54640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:48.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.503+0000 7fe7b9d54640 10 monclient: _renew_subs 2024-03-25T19:44:48.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.503+0000 7fe7b9d54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:48.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.503+0000 7fe7b9d54640 1 -- 172.21.15.107:0/1218447246 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b41e3cd0 con 0x7fe7b40b4d60 2024-03-25T19:44:48.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.503+0000 7fe7b9d54640 1 librados: init done 2024-03-25T19:44:48.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.504+0000 7fe7b9d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:48.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.504+0000 7fe7b9d54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:48.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.504+0000 7fe7b9d54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:48.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.504+0000 7fe7b9d54640 1 -- 172.21.15.107:0/1218447246 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b4140410 con 0x7fe7b40b4d60 2024-03-25T19:44:48.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.809+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7a8001270 con 0x7fe7b40b4d60 2024-03-25T19:44:48.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.809+0000 7fe79bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.809+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe7a8003960 con 0x7fe7b40b4d60 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.809+0000 7fe79bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.809+0000 7fe79bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.809+0000 7fe79bfff640 10 monclient: dump: 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:48.810 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:48.811 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:48.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.809+0000 7fe7b1ffb640 4 set_mon_vals no callback set 2024-03-25T19:44:48.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.810+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fe7a80199b0 con 0x7fe7b40b4d60 2024-03-25T19:44:48.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.811+0000 7fe7b37fe640 10 monclient: get_auth_request con 0x7fe7a006e0c0 auth_method 0 2024-03-25T19:44:48.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.811+0000 7fe79bfff640 10 client.4626.objecter ms_handle_connect 0x7fe7a006e0c0 2024-03-25T19:44:48.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.816+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fe7a80535d0 con 0x7fe7b40b4d60 2024-03-25T19:44:48.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.816+0000 7fe79bfff640 10 client.4626.objecter ms_dispatch 0x7fe7b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:48.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.816+0000 7fe79bfff640 3 client.4626.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:48.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.816+0000 7fe79bfff640 3 client.4626.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:48.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.816+0000 7fe79bfff640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-25T19:44:48.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.816+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7a804ef10 con 0x7fe7b40b4d60 2024-03-25T19:44:48.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.816+0000 7fe79bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:48.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:48.816+0000 7fe79bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.069+0000 7fe7b9d54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.069+0000 7fe7b9d54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.069+0000 7fe7b9d54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:49.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.069+0000 7fe7b9d54640 1 -- 172.21.15.107:0/1218447246 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7b40627b0 con 0x7fe7b40b4d60 2024-03-25T19:44:49.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.089+0000 7fe79bfff640 1 -- 172.21.15.107:0/1218447246 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fe7a8051010 con 0x7fe7b40b4d60 2024-03-25T19:44:49.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.090+0000 7fe79bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:49.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.090+0000 7fe79bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:49.090 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:44:49.090 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:49.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 10 librados: watch_flush enter 2024-03-25T19:44:49.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 10 librados: watch_flush exit 2024-03-25T19:44:49.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-25T19:44:49.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 10 client.4626.objecter successfully canceled tick 2024-03-25T19:44:49.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 1 -- 172.21.15.107:0/1218447246 >> v1:172.21.15.195:6800/2984408421 conn(0x7fe7a006e0c0 legacy=0x7fe7a0070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:49.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 10 monclient: shutdown 2024-03-25T19:44:49.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 1 -- 172.21.15.107:0/1218447246 >> v1:172.21.15.107:6789/0 conn(0x7fe7b40b4d60 legacy=0x7fe7b41406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 1 -- 172.21.15.107:0/1218447246 shutdown_connections 2024-03-25T19:44:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 1 -- 172.21.15.107:0/1218447246 >> 172.21.15.107:0/1218447246 conn(0x7fe7b40617a0 msgr2=0x7fe7b413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 1 -- 172.21.15.107:0/1218447246 shutdown_connections 2024-03-25T19:44:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 1 -- 172.21.15.107:0/1218447246 wait complete. 2024-03-25T19:44:49.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:49.092+0000 7fe7b9d54640 1 librados: shutdown 2024-03-25T19:44:49.105 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:55.912 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:56.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.029+0000 7f94dee89640 10 monclient: get_monmap_and_config 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.029+0000 7f94dee89640 10 monclient: build_initial_monmap 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.029+0000 7f94dee89640 10 monclient: monmap: 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:44:56.030076+0000 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:44:56.030076+0000 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:44:56.029 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:56.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 1 Processor -- start 2024-03-25T19:44:56.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 1 -- start start 2024-03-25T19:44:56.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 10 monclient: init 2024-03-25T19:44:56.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:56.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:56.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 10 monclient(hunting): picked mon.noname-c con 0x7f94d80b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:56.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 10 monclient(hunting): picked mon.noname-b con 0x7f94d80b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:56.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 10 monclient(hunting): picked mon.noname-a con 0x7f94d814be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:56.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d80af940 con 0x7f94d814be90 2024-03-25T19:44:56.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d80b0a70 con 0x7f94d80b4d60 2024-03-25T19:44:56.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d80b0d50 con 0x7f94d80b1730 2024-03-25T19:44:56.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:56.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.030+0000 7f94dee89640 10 monclient(hunting): authenticate will time out at 5088.371649s 2024-03-25T19:44:56.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.037+0000 7f94dcc25640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f94d80b4d60 0x7f94d813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:37672/0 (socket says 172.21.15.107:37672) 2024-03-25T19:44:56.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.037+0000 7f94dcc25640 1 -- 172.21.15.107:0/30685017 learned_addr learned my addr 172.21.15.107:0/30685017 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:56.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.046+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978150384 0 0) 0x7f94d80b0a70 con 0x7f94d80b4d60 2024-03-25T19:44:56.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.046+0000 7f94d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:56.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.046+0000 7f94d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:56.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.046+0000 7f94d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:56.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.046+0000 7f94d77fe640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:44:56.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.046+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94c8003520 con 0x7f94d80b4d60 2024-03-25T19:44:56.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.053+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705957829 0 0) 0x7f94d80af940 con 0x7f94d814be90 2024-03-25T19:44:56.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.053+0000 7f94d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:56.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.053+0000 7f94d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:56.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.053+0000 7f94d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:56.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.053+0000 7f94d77fe640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:44:56.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.053+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d80b0a70 con 0x7f94d814be90 2024-03-25T19:44:56.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.058+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882641652 0 0) 0x7f94d80b0d50 con 0x7f94d80b1730 2024-03-25T19:44:56.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.058+0000 7f94d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:56.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.058+0000 7f94d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:56.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.058+0000 7f94d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:56.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.058+0000 7f94d77fe640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:44:56.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.058+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d80af940 con 0x7f94d80b1730 2024-03-25T19:44:56.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2696871916 0 0) 0x7f94c8003520 con 0x7f94d80b4d60 2024-03-25T19:44:56.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 >> v1:172.21.15.107:6790/0 conn(0x7f94d80b1730 legacy=0x7f94d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 >> v1:172.21.15.107:6789/0 conn(0x7f94d814be90 legacy=0x7f94d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:56.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 1 monclient: found mon.noname-c 2024-03-25T19:44:56.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:56.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:44:56.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d80b1030 con 0x7f94d80b4d60 2024-03-25T19:44:56.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f94cc001690 con 0x7f94d80b4d60 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94d77fe640 10 monclient: dump: 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:56.065 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:56.066 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:56.066 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:56.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94dee89640 5 monclient: authenticate success, global_id 4621 2024-03-25T19:44:56.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.064+0000 7f94dee89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:56.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.071+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94cc0017f0 con 0x7f94d80b4d60 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.071+0000 7f94d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.071+0000 7f94d77fe640 1 -- 172.21.15.107:0/30685017 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f94cc001c00 con 0x7f94d80b4d60 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.071+0000 7f94d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.071+0000 7f94d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.071+0000 7f94d77fe640 10 monclient: dump: 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:56.071 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:56.072 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:56.072 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:56.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.071+0000 7f94dee89640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:56.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.071+0000 7f94dee89640 4 set_mon_vals no callback set 2024-03-25T19:44:56.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.072+0000 7f94dee89640 10 monclient: shutdown 2024-03-25T19:44:56.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.072+0000 7f94dee89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:56.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.072+0000 7f94dee89640 1 -- 172.21.15.107:0/30685017 >> v1:172.21.15.195:6789/0 conn(0x7f94d80b4d60 legacy=0x7f94d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.072+0000 7f94dee89640 1 -- 172.21.15.107:0/30685017 shutdown_connections 2024-03-25T19:44:56.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.072+0000 7f94dee89640 1 -- 172.21.15.107:0/30685017 >> 172.21.15.107:0/30685017 conn(0x7f94d80617a0 msgr2=0x7f94d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:56.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.072+0000 7f94dee89640 1 -- 172.21.15.107:0/30685017 shutdown_connections 2024-03-25T19:44:56.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.072+0000 7f94dee89640 1 -- 172.21.15.107:0/30685017 wait complete. 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.073+0000 7f94dee89640 10 monclient: build_initial_monmap 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.073+0000 7f94dee89640 10 monclient: monmap: 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.073+0000 7f94dee89640 1 librados: starting msgr at 2024-03-25T19:44:56.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.073+0000 7f94dee89640 1 librados: starting objecter 2024-03-25T19:44:56.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.073+0000 7f94dee89640 1 Processor -- start 2024-03-25T19:44:56.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.073+0000 7f94dee89640 1 -- start start 2024-03-25T19:44:56.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.073+0000 7f94dee89640 1 librados: setting wanted keys 2024-03-25T19:44:56.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.073+0000 7f94dee89640 1 librados: calling monclient init 2024-03-25T19:44:56.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.073+0000 7f94dee89640 10 monclient: init 2024-03-25T19:44:56.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.074+0000 7f94dee89640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:56.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.074+0000 7f94dee89640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:56.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.074+0000 7f94dee89640 10 monclient(hunting): picked mon.noname-c con 0x7f94d80b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:44:56.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.074+0000 7f94dee89640 10 monclient(hunting): picked mon.noname-b con 0x7f94d80b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:44:56.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.074+0000 7f94dee89640 10 monclient(hunting): picked mon.noname-a con 0x7f94d814be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:44:56.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.074+0000 7f94dee89640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d8138670 con 0x7f94d814be90 2024-03-25T19:44:56.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.074+0000 7f94dee89640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d8138950 con 0x7f94d80b4d60 2024-03-25T19:44:56.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.074+0000 7f94dee89640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d8140620 con 0x7f94d80b1730 2024-03-25T19:44:56.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.074+0000 7f94dee89640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:56.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.074+0000 7f94dee89640 10 monclient(hunting): authenticate will time out at 5088.415474s 2024-03-25T19:44:56.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.078+0000 7f94dd426640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f94d814be90 0x7f94d81e2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57848/0 (socket says 172.21.15.107:57848) 2024-03-25T19:44:56.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.078+0000 7f94dd426640 1 -- 172.21.15.107:0/3009139455 learned_addr learned my addr 172.21.15.107:0/3009139455 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:44:56.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.078+0000 7f94d57fa640 10 client.?.objecter ms_handle_connect 0x7f94d814be90 2024-03-25T19:44:56.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.078+0000 7f94d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:56.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.089+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34574607 0 0) 0x7f94d8138670 con 0x7f94d814be90 2024-03-25T19:44:56.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.089+0000 7f94d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:56.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.089+0000 7f94d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:56.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.089+0000 7f94d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:56.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.089+0000 7f94d57fa640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:44:56.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.089+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94bc004eb0 con 0x7f94d814be90 2024-03-25T19:44:56.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.090+0000 7f94d57fa640 10 client.?.objecter ms_handle_connect 0x7f94d80b4d60 2024-03-25T19:44:56.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.090+0000 7f94d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:56.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.100+0000 7f94d57fa640 10 client.?.objecter ms_handle_connect 0x7f94d80b1730 2024-03-25T19:44:56.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.100+0000 7f94d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:56.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.103+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2320410692 0 0) 0x7f94bc004eb0 con 0x7f94d814be90 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.103+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94d8138670 con 0x7f94d814be90 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.103+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f94d0001c90 con 0x7f94d814be90 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.103+0000 7f94d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.103+0000 7f94d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.103+0000 7f94d57fa640 10 monclient(hunting): dump: 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:56.103 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:56.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.103+0000 7f94d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:56.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.108+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452933819 0 0) 0x7f94d8138950 con 0x7f94d80b4d60 2024-03-25T19:44:56.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.108+0000 7f94d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:56.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.108+0000 7f94d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:56.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.108+0000 7f94d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:56.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.108+0000 7f94d57fa640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:44:56.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.108+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94bc004eb0 con 0x7f94d80b4d60 2024-03-25T19:44:56.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.111+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836457425 0 0) 0x7f94d8140620 con 0x7f94d80b1730 2024-03-25T19:44:56.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.111+0000 7f94d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:56.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.111+0000 7f94d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:56.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.111+0000 7f94d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:56.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.111+0000 7f94d57fa640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:44:56.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.111+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d8138950 con 0x7f94d80b1730 2024-03-25T19:44:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3285867524 0 0) 0x7f94d8138670 con 0x7f94d814be90 2024-03-25T19:44:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 >> v1:172.21.15.107:6790/0 conn(0x7f94d80b1730 legacy=0x7f94d8144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 >> v1:172.21.15.195:6789/0 conn(0x7f94d80b4d60 legacy=0x7f94d8140a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94d57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94d57fa640 1 monclient: found mon.a 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94d57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d8137bc0 con 0x7f94d814be90 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94d57fa640 10 monclient: _finish_auth 0 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94d57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94dee89640 5 monclient: authenticate success, global_id 4632 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94dee89640 10 monclient: _renew_subs 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94dee89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94dee89640 1 -- 172.21.15.107:0/3009139455 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d8142b90 con 0x7f94d814be90 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94dee89640 10 client.4632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94dee89640 10 monclient: _renew_subs 2024-03-25T19:44:56.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94dee89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:56.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94dee89640 1 -- 172.21.15.107:0/3009139455 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d81377a0 con 0x7f94d814be90 2024-03-25T19:44:56.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.120+0000 7f94dee89640 1 librados: init done 2024-03-25T19:44:56.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.121+0000 7f94dee89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:56.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.121+0000 7f94dee89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:56.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.121+0000 7f94dee89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:56.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.121+0000 7f94dee89640 1 -- 172.21.15.107:0/3009139455 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d80b3880 con 0x7f94d814be90 2024-03-25T19:44:56.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.417+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94d0003970 con 0x7f94d814be90 2024-03-25T19:44:56.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.417+0000 7f94d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:56.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.417+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f94d0003d40 con 0x7f94d814be90 2024-03-25T19:44:56.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.417+0000 7f94d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:56.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.417+0000 7f94d57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:56.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.417+0000 7f94d57fa640 10 monclient: dump: 2024-03-25T19:44:56.419 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:44:56.419 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:44:56.419 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:56.419 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:44:56.419 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:56.419 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:44:56.419 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:44:56.420 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:44:56.420 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:44:56.420 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:44:56.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.417+0000 7f94d6ffd640 4 set_mon_vals no callback set 2024-03-25T19:44:56.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.418+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f94d0003f60 con 0x7f94d814be90 2024-03-25T19:44:56.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.419+0000 7f94dcc25640 10 monclient: get_auth_request con 0x7f94bc06de80 auth_method 0 2024-03-25T19:44:56.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.419+0000 7f94d57fa640 10 client.4632.objecter ms_handle_connect 0x7f94bc06de80 2024-03-25T19:44:56.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.425+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f94d0054a70 con 0x7f94d814be90 2024-03-25T19:44:56.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.425+0000 7f94d57fa640 10 client.4632.objecter ms_dispatch 0x7f94d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:56.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.425+0000 7f94d57fa640 3 client.4632.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:56.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.425+0000 7f94d57fa640 3 client.4632.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:56.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.425+0000 7f94d57fa640 20 client.4632.objecter dump_active .. 0 homeless 2024-03-25T19:44:56.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.425+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94d0050580 con 0x7f94d814be90 2024-03-25T19:44:56.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.425+0000 7f94d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:56.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.425+0000 7f94d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:56.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.650+0000 7f94dee89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:56.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.650+0000 7f94dee89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:56.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.650+0000 7f94dee89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:44:56.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.650+0000 7f94dee89640 1 -- 172.21.15.107:0/3009139455 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94d80627b0 con 0x7f94d814be90 2024-03-25T19:44:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.667+0000 7f94d57fa640 1 -- 172.21.15.107:0/3009139455 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f94d00524b0 con 0x7f94d814be90 2024-03-25T19:44:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.667+0000 7f94d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:56.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.667+0000 7f94d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:56.667 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:44:56.667 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:56.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 10 librados: watch_flush enter 2024-03-25T19:44:56.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 10 librados: watch_flush exit 2024-03-25T19:44:56.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 20 client.4632.objecter shutdown clearing up homeless session... 2024-03-25T19:44:56.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 10 client.4632.objecter successfully canceled tick 2024-03-25T19:44:56.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 1 -- 172.21.15.107:0/3009139455 >> v1:172.21.15.195:6800/2984408421 conn(0x7f94bc06de80 legacy=0x7f94bc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 10 monclient: shutdown 2024-03-25T19:44:56.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:56.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 1 -- 172.21.15.107:0/3009139455 >> v1:172.21.15.107:6789/0 conn(0x7f94d814be90 legacy=0x7f94d81e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:56.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 1 -- 172.21.15.107:0/3009139455 shutdown_connections 2024-03-25T19:44:56.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 1 -- 172.21.15.107:0/3009139455 >> 172.21.15.107:0/3009139455 conn(0x7f94d80617a0 msgr2=0x7f94d80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:56.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 1 -- 172.21.15.107:0/3009139455 shutdown_connections 2024-03-25T19:44:56.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 1 -- 172.21.15.107:0/3009139455 wait complete. 2024-03-25T19:44:56.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:44:56.670+0000 7f94dee89640 1 librados: shutdown 2024-03-25T19:44:56.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:03.488 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:03.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.605+0000 7efec5295640 10 monclient: get_monmap_and_config 2024-03-25T19:45:03.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.605+0000 7efec5295640 10 monclient: build_initial_monmap 2024-03-25T19:45:03.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.606+0000 7efec5295640 10 monclient: monmap: 2024-03-25T19:45:03.606 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:03.606 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:03.606 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:45:03.606976+0000 2024-03-25T19:45:03.606 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:45:03.606976+0000 2024-03-25T19:45:03.606 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:03.606 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:03.606 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:03.606 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:45:03.606 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:45:03.606 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:03.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.606+0000 7efec5295640 1 Processor -- start 2024-03-25T19:45:03.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.606+0000 7efec5295640 1 -- start start 2024-03-25T19:45:03.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.606+0000 7efec5295640 10 monclient: init 2024-03-25T19:45:03.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.607+0000 7efec5295640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:03.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.607+0000 7efec5295640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:03.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.607+0000 7efec5295640 10 monclient(hunting): picked mon.noname-a con 0x7efec006ab80 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:03.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.607+0000 7efec5295640 10 monclient(hunting): picked mon.noname-b con 0x7efec006af60 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:03.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.607+0000 7efec5295640 10 monclient(hunting): picked mon.noname-c con 0x7efec00b00b0 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:03.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.607+0000 7efec5295640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec00b0500 con 0x7efec006ab80 2024-03-25T19:45:03.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.607+0000 7efec5295640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec013ffa0 con 0x7efec00b00b0 2024-03-25T19:45:03.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.607+0000 7efec5295640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec00af520 con 0x7efec006af60 2024-03-25T19:45:03.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.607+0000 7efec5295640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:03.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.607+0000 7efec5295640 10 monclient(hunting): authenticate will time out at 5095.948299s 2024-03-25T19:45:03.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.617+0000 7efebf7fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7efec00b00b0 0x7efec014dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:44832/0 (socket says 172.21.15.107:44832) 2024-03-25T19:45:03.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.617+0000 7efebf7fe640 1 -- 172.21.15.107:0/2482840869 learned_addr learned my addr 172.21.15.107:0/2482840869 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:03.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.639+0000 7efebdffb640 1 -- 172.21.15.107:0/2482840869 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752954043 0 0) 0x7efec013ffa0 con 0x7efec00b00b0 2024-03-25T19:45:03.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.639+0000 7efebdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.639+0000 7efebdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.639+0000 7efebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.639+0000 7efebdffb640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:45:03.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.639+0000 7efebdffb640 1 -- 172.21.15.107:0/2482840869 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea8003520 con 0x7efec00b00b0 2024-03-25T19:45:03.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.646+0000 7efebdffb640 1 -- 172.21.15.107:0/2482840869 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192801769 0 0) 0x7efec00af520 con 0x7efec006af60 2024-03-25T19:45:03.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.646+0000 7efebdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.646+0000 7efebdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.646+0000 7efebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.646+0000 7efebdffb640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:45:03.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.646+0000 7efebdffb640 1 -- 172.21.15.107:0/2482840869 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efec013ffa0 con 0x7efec006af60 2024-03-25T19:45:03.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 1 -- 172.21.15.107:0/2482840869 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2932816819 0 0) 0x7efea8003520 con 0x7efec00b00b0 2024-03-25T19:45:03.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 1 -- 172.21.15.107:0/2482840869 >> v1:172.21.15.107:6790/0 conn(0x7efec006af60 legacy=0x7efec013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 1 -- 172.21.15.107:0/2482840869 >> v1:172.21.15.107:6789/0 conn(0x7efec006ab80 legacy=0x7efec00ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:03.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 1 monclient: found mon.noname-c 2024-03-25T19:45:03.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:03.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:45:03.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 1 -- 172.21.15.107:0/2482840869 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec0072890 con 0x7efec00b00b0 2024-03-25T19:45:03.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 1 -- 172.21.15.107:0/2482840869 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7efeb8001a10 con 0x7efec00b00b0 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efec5295640 5 monclient: authenticate success, global_id 4627 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efec5295640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efebdffb640 10 monclient: dump: 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:03.654 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:03.655 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:03.655 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:03.655 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:03.655 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.653+0000 7efec5295640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:03.662 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.662+0000 7efebdffb640 1 -- 172.21.15.107:0/2482840869 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeb8001b70 con 0x7efec00b00b0 2024-03-25T19:45:03.662 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.662+0000 7efebdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.662+0000 7efebdffb640 1 -- 172.21.15.107:0/2482840869 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7efeb8001f80 con 0x7efec00b00b0 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.662+0000 7efebdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.662+0000 7efebdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.662+0000 7efebdffb640 10 monclient: dump: 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:03.663 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.662+0000 7efec5295640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:03.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.662+0000 7efec5295640 4 set_mon_vals no callback set 2024-03-25T19:45:03.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.663+0000 7efec5295640 10 monclient: shutdown 2024-03-25T19:45:03.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.663+0000 7efec5295640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.663+0000 7efec5295640 1 -- 172.21.15.107:0/2482840869 >> v1:172.21.15.195:6789/0 conn(0x7efec00b00b0 legacy=0x7efec014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.663+0000 7efec5295640 1 -- 172.21.15.107:0/2482840869 shutdown_connections 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.663+0000 7efec5295640 1 -- 172.21.15.107:0/2482840869 >> 172.21.15.107:0/2482840869 conn(0x7efec00b4a80 msgr2=0x7efec00b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.663+0000 7efec5295640 1 -- 172.21.15.107:0/2482840869 shutdown_connections 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.663+0000 7efec5295640 1 -- 172.21.15.107:0/2482840869 wait complete. 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.664+0000 7efec5295640 10 monclient: build_initial_monmap 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.664+0000 7efec5295640 10 monclient: monmap: 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:45:03.667 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.665+0000 7efec5295640 1 librados: starting msgr at 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.665+0000 7efec5295640 1 librados: starting objecter 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.665+0000 7efec5295640 1 Processor -- start 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.665+0000 7efec5295640 1 -- start start 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.665+0000 7efec5295640 1 librados: setting wanted keys 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.665+0000 7efec5295640 1 librados: calling monclient init 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.665+0000 7efec5295640 10 monclient: init 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.666+0000 7efec5295640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:03.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.666+0000 7efec5295640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:03.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.666+0000 7efec5295640 10 monclient(hunting): picked mon.noname-b con 0x7efec006ab80 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:03.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.666+0000 7efec5295640 10 monclient(hunting): picked mon.noname-a con 0x7efec00b00b0 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:03.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.666+0000 7efec5295640 10 monclient(hunting): picked mon.noname-c con 0x7efec0137e80 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:03.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.666+0000 7efec5295640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec0143790 con 0x7efec00b00b0 2024-03-25T19:45:03.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.666+0000 7efec5295640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec0145780 con 0x7efec006ab80 2024-03-25T19:45:03.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.666+0000 7efec5295640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec0143460 con 0x7efec0137e80 2024-03-25T19:45:03.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.666+0000 7efec5295640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:03.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.666+0000 7efec5295640 10 monclient(hunting): authenticate will time out at 5096.007005s 2024-03-25T19:45:03.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.683+0000 7efebeffd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7efec006ab80 0x7efec0146180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:44858/0 (socket says 172.21.15.107:44858) 2024-03-25T19:45:03.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.683+0000 7efebeffd640 1 -- 172.21.15.107:0/957050690 learned_addr learned my addr 172.21.15.107:0/957050690 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:03.684 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.684+0000 7efea77fe640 10 client.?.objecter ms_handle_connect 0x7efec006ab80 2024-03-25T19:45:03.684 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.684+0000 7efea77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:03.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.693+0000 7efea77fe640 10 client.?.objecter ms_handle_connect 0x7efec00b00b0 2024-03-25T19:45:03.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.693+0000 7efea77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:03.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.697+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252487712 0 0) 0x7efec0145780 con 0x7efec006ab80 2024-03-25T19:45:03.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.697+0000 7efea77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.697+0000 7efea77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.697+0000 7efea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.697+0000 7efea77fe640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:45:03.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.697+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeac004eb0 con 0x7efec006ab80 2024-03-25T19:45:03.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.698+0000 7efea77fe640 10 client.?.objecter ms_handle_connect 0x7efec0137e80 2024-03-25T19:45:03.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.698+0000 7efea77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:03.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.707+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682632824 0 0) 0x7efec0143460 con 0x7efec0137e80 2024-03-25T19:45:03.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.707+0000 7efea77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.707+0000 7efea77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.707+0000 7efea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.707+0000 7efea77fe640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:45:03.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.707+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efec0145780 con 0x7efec0137e80 2024-03-25T19:45:03.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.711+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115914607 0 0) 0x7efec0143790 con 0x7efec00b00b0 2024-03-25T19:45:03.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.711+0000 7efea77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.711+0000 7efea77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.711+0000 7efea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.711+0000 7efea77fe640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:45:03.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.711+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efec0143460 con 0x7efec00b00b0 2024-03-25T19:45:03.714 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.714+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4006535094 0 0) 0x7efeac004eb0 con 0x7efec006ab80 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.714+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efec0143790 con 0x7efec006ab80 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.715+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7efeb0001a30 con 0x7efec006ab80 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.715+0000 7efea77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.715+0000 7efea77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.715+0000 7efea77fe640 10 monclient(hunting): dump: 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:03.715 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:03.716 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:03.716 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:03.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.715+0000 7efea77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:03.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4261076157 0 0) 0x7efec0143790 con 0x7efec006ab80 2024-03-25T19:45:03.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 >> v1:172.21.15.107:6790/0 conn(0x7efec0137e80 legacy=0x7efec0144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 >> v1:172.21.15.107:6789/0 conn(0x7efec00b00b0 legacy=0x7efec0143c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efea77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efea77fe640 1 monclient: found mon.b 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efea77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efea77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec01e6540 con 0x7efec006ab80 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efea77fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efea77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efea77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efec5295640 5 monclient: authenticate success, global_id 4630 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efec5295640 10 monclient: _renew_subs 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efec5295640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efec5295640 1 -- 172.21.15.107:0/957050690 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec01e3510 con 0x7efec006ab80 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efec5295640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efec5295640 10 monclient: _renew_subs 2024-03-25T19:45:03.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efec5295640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:45:03.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efec5295640 1 -- 172.21.15.107:0/957050690 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efec01e3a40 con 0x7efec006ab80 2024-03-25T19:45:03.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.723+0000 7efec5295640 1 librados: init done 2024-03-25T19:45:03.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.724+0000 7efec5295640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:03.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.724+0000 7efec5295640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:03.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.724+0000 7efec5295640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:45:03.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:03.724+0000 7efec5295640 1 -- 172.21.15.107:0/957050690 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec0071c50 con 0x7efec006ab80 2024-03-25T19:45:04.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.010+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeb0001c70 con 0x7efec006ab80 2024-03-25T19:45:04.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.010+0000 7efea77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:04.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.010+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7efeb0003a60 con 0x7efec006ab80 2024-03-25T19:45:04.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.010+0000 7efea77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:04.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.010+0000 7efea77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:04.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.010+0000 7efea77fe640 10 monclient: dump: 2024-03-25T19:45:04.011 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:04.011 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:04.011 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:04.011 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:04.011 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:04.011 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:04.011 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:04.011 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:04.011 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:04.011 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:04.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.010+0000 7efebd7fa640 4 set_mon_vals no callback set 2024-03-25T19:45:04.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.010+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7efeb0003c80 con 0x7efec006ab80 2024-03-25T19:45:04.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.011+0000 7efebe7fc640 10 monclient: get_auth_request con 0x7efeac06ddf0 auth_method 0 2024-03-25T19:45:04.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.011+0000 7efea77fe640 10 client.4630.objecter ms_handle_connect 0x7efeac06ddf0 2024-03-25T19:45:04.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.015+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7efeb0054860 con 0x7efec006ab80 2024-03-25T19:45:04.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.015+0000 7efea77fe640 10 client.4630.objecter ms_dispatch 0x7efec0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:04.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.015+0000 7efea77fe640 3 client.4630.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:04.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.015+0000 7efea77fe640 3 client.4630.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:04.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.015+0000 7efea77fe640 20 client.4630.objecter dump_active .. 0 homeless 2024-03-25T19:45:04.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.015+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efeb0050370 con 0x7efec006ab80 2024-03-25T19:45:04.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.015+0000 7efea77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:04.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.015+0000 7efea77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:04.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.260+0000 7efec5295640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:04.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.260+0000 7efec5295640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:04.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.260+0000 7efec5295640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:45:04.260 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.260+0000 7efec5295640 1 -- 172.21.15.107:0/957050690 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efec00b0be0 con 0x7efec006ab80 2024-03-25T19:45:04.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.375+0000 7efea77fe640 1 -- 172.21.15.107:0/957050690 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7efeb00522a0 con 0x7efec006ab80 2024-03-25T19:45:04.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.375+0000 7efea77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:04.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.375+0000 7efea77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:04.375 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:45:04.376 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:04.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 10 librados: watch_flush enter 2024-03-25T19:45:04.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 10 librados: watch_flush exit 2024-03-25T19:45:04.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 20 client.4630.objecter shutdown clearing up homeless session... 2024-03-25T19:45:04.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 10 client.4630.objecter successfully canceled tick 2024-03-25T19:45:04.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 1 -- 172.21.15.107:0/957050690 >> v1:172.21.15.195:6800/2984408421 conn(0x7efeac06ddf0 legacy=0x7efeac0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:04.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 10 monclient: shutdown 2024-03-25T19:45:04.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:04.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 1 -- 172.21.15.107:0/957050690 >> v1:172.21.15.195:6789/0 conn(0x7efec006ab80 legacy=0x7efec0146180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:04.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 1 -- 172.21.15.107:0/957050690 shutdown_connections 2024-03-25T19:45:04.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 1 -- 172.21.15.107:0/957050690 >> 172.21.15.107:0/957050690 conn(0x7efec00b4a80 msgr2=0x7efec00b2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:04.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 1 -- 172.21.15.107:0/957050690 shutdown_connections 2024-03-25T19:45:04.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 1 -- 172.21.15.107:0/957050690 wait complete. 2024-03-25T19:45:04.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:04.378+0000 7efec5295640 1 librados: shutdown 2024-03-25T19:45:04.391 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:11.197 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:11.297 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.298+0000 7f9186939640 10 monclient: get_monmap_and_config 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.298+0000 7f9186939640 10 monclient: build_initial_monmap 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.298+0000 7f9186939640 10 monclient: monmap: 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:45:11.299132+0000 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:45:11.299132+0000 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:45:11.298 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.298+0000 7f9186939640 1 Processor -- start 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.298+0000 7f9186939640 1 -- start start 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.298+0000 7f9186939640 10 monclient: init 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f9186939640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f9186939640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f9186939640 10 monclient(hunting): picked mon.noname-a con 0x7f91800b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f9186939640 10 monclient(hunting): picked mon.noname-c con 0x7f91800b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f9186939640 10 monclient(hunting): picked mon.noname-b con 0x7f918014be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f9186939640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91800af940 con 0x7f91800b4d60 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f9186939640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91800b0a70 con 0x7f91800b1730 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f9186939640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91800b0d50 con 0x7f918014be90 2024-03-25T19:45:11.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f9186939640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:11.301 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f9186939640 10 monclient(hunting): authenticate will time out at 5103.640258s 2024-03-25T19:45:11.301 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f917ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f91800b4d60 0x7f918013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58374/0 (socket says 172.21.15.107:58374) 2024-03-25T19:45:11.301 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.299+0000 7f917ffff640 1 -- 172.21.15.107:0/1616855417 learned_addr learned my addr 172.21.15.107:0/1616855417 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:11.308 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:09:41.625 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.315 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:09:41.631 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.336+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365417924 0 0) 0x7f91800b0d50 con 0x7f918014be90 2024-03-25T19:45:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.336+0000 7f917effd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.336+0000 7f917effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.336+0000 7f917effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.336+0000 7f917effd640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:45:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.336+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9168003520 con 0x7f918014be90 2024-03-25T19:45:11.344 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:09:41.659 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.355 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:09:41.641 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.360+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825548179 0 0) 0x7f91800b0a70 con 0x7f91800b1730 2024-03-25T19:45:11.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.360+0000 7f917effd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.360+0000 7f917effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.360+0000 7f917effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.360+0000 7f917effd640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:45:11.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.360+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91800b0d50 con 0x7f91800b1730 2024-03-25T19:45:11.363 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:09:41.679 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.366 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:09:41.653 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.369+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402481140 0 0) 0x7f91800af940 con 0x7f91800b4d60 2024-03-25T19:45:11.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.369+0000 7f917effd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.369+0000 7f917effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.369+0000 7f917effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.369+0000 7f917effd640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:45:11.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.369+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91800b0a70 con 0x7f91800b4d60 2024-03-25T19:45:11.383 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:09:41.699 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3109278282 0 0) 0x7f91800b0d50 con 0x7f91800b1730 2024-03-25T19:45:11.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 >> v1:172.21.15.107:6790/0 conn(0x7f918014be90 legacy=0x7f918014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 >> v1:172.21.15.107:6789/0 conn(0x7f91800b4d60 legacy=0x7f918013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 1 monclient: found mon.noname-c 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91800b1030 con 0x7f91800b1730 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 10 monclient: _finish_auth 0 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f9174001670 con 0x7f91800b1730 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f917effd640 10 monclient: dump: 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:11.399 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:11.400 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:11.400 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:11.400 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:11.400 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:11.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f9186939640 5 monclient: authenticate success, global_id 4633 2024-03-25T19:45:11.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.398+0000 7f9186939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:11.408 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:09:41.695 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.418 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:09:41.734 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:11.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.438+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91740017d0 con 0x7f91800b1730 2024-03-25T19:45:11.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.438+0000 7f917effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:11.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.438+0000 7f917effd640 1 -- 172.21.15.107:0/1616855417 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f9174001be0 con 0x7f91800b1730 2024-03-25T19:45:11.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.438+0000 7f917effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:11.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.438+0000 7f917effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.438+0000 7f917effd640 10 monclient: dump: 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.438+0000 7f9186939640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:11.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.438+0000 7f9186939640 4 set_mon_vals no callback set 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.439+0000 7f9186939640 10 monclient: shutdown 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.439+0000 7f9186939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.439+0000 7f9186939640 1 -- 172.21.15.107:0/1616855417 >> v1:172.21.15.195:6789/0 conn(0x7f91800b1730 legacy=0x7f91800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.439+0000 7f9186939640 1 -- 172.21.15.107:0/1616855417 shutdown_connections 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.439+0000 7f9186939640 1 -- 172.21.15.107:0/1616855417 >> 172.21.15.107:0/1616855417 conn(0x7f91800617a0 msgr2=0x7f9180061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.439+0000 7f9186939640 1 -- 172.21.15.107:0/1616855417 shutdown_connections 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.439+0000 7f9186939640 1 -- 172.21.15.107:0/1616855417 wait complete. 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.440+0000 7f9186939640 10 monclient: build_initial_monmap 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.440+0000 7f9186939640 10 monclient: monmap: 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:11.441 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:45:11.442 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:11.442 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:45:11.442 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:45:11.442 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:11.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.440+0000 7f9186939640 1 librados: starting msgr at 2024-03-25T19:45:11.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.440+0000 7f9186939640 1 librados: starting objecter 2024-03-25T19:45:11.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.441+0000 7f9186939640 1 Processor -- start 2024-03-25T19:45:11.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.441+0000 7f9186939640 1 -- start start 2024-03-25T19:45:11.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.441+0000 7f9186939640 1 librados: setting wanted keys 2024-03-25T19:45:11.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.441+0000 7f9186939640 1 librados: calling monclient init 2024-03-25T19:45:11.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.441+0000 7f9186939640 10 monclient: init 2024-03-25T19:45:11.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.442+0000 7f9186939640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:11.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.442+0000 7f9186939640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:11.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.442+0000 7f9186939640 10 monclient(hunting): picked mon.noname-a con 0x7f91800b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:11.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.442+0000 7f9186939640 10 monclient(hunting): picked mon.noname-b con 0x7f91800b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:11.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.442+0000 7f9186939640 10 monclient(hunting): picked mon.noname-c con 0x7f918014be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:11.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.442+0000 7f9186939640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9180141ec0 con 0x7f91800b1730 2024-03-25T19:45:11.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.442+0000 7f9186939640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9180142110 con 0x7f91800b4d60 2024-03-25T19:45:11.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.442+0000 7f9186939640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91801d7960 con 0x7f918014be90 2024-03-25T19:45:11.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.442+0000 7f9186939640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:11.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.442+0000 7f9186939640 10 monclient(hunting): authenticate will time out at 5103.783164s 2024-03-25T19:45:11.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.470+0000 7f9184ed6640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f918014be90 0x7f9180142c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34140/0 (socket says 172.21.15.107:34140) 2024-03-25T19:45:11.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.470+0000 7f9184ed6640 1 -- 172.21.15.107:0/1560351512 learned_addr learned my addr 172.21.15.107:0/1560351512 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:11.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.475+0000 7f917cff9640 10 client.?.objecter ms_handle_connect 0x7f918014be90 2024-03-25T19:45:11.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.475+0000 7f917cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:11.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.485+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237795076 0 0) 0x7f91801d7960 con 0x7f918014be90 2024-03-25T19:45:11.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.485+0000 7f917cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.485+0000 7f917cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.485+0000 7f917cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.485+0000 7f917cff9640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:45:11.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.485+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f916c004eb0 con 0x7f918014be90 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.495+0000 7f917cff9640 10 client.?.objecter ms_handle_connect 0x7f91800b4d60 2024-03-25T19:45:11.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.495+0000 7f917cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:11.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.500+0000 7f917cff9640 10 client.?.objecter ms_handle_connect 0x7f91800b1730 2024-03-25T19:45:11.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.500+0000 7f917cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:11.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.514+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472509699 0 0) 0x7f9180141ec0 con 0x7f91800b1730 2024-03-25T19:45:11.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.514+0000 7f917cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.514+0000 7f917cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.514+0000 7f917cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.514+0000 7f917cff9640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:45:11.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.514+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91801d7960 con 0x7f91800b1730 2024-03-25T19:45:11.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669549502 0 0) 0x7f916c004eb0 con 0x7f918014be90 2024-03-25T19:45:11.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9180141ec0 con 0x7f918014be90 2024-03-25T19:45:11.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f9178001c30 con 0x7f918014be90 2024-03-25T19:45:11.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:11.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:11.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 10 monclient(hunting): dump: 2024-03-25T19:45:11.523 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:11.523 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:11.523 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807896059 0 0) 0x7f9180142110 con 0x7f91800b4d60 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:45:11.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.523+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f916c004eb0 con 0x7f91800b4d60 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.537+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2517667204 0 0) 0x7f91801d7960 con 0x7f91800b1730 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.537+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9180142110 con 0x7f91800b1730 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.537+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f9170001870 con 0x7f91800b1730 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.537+0000 7f917cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.537+0000 7f917cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.537+0000 7f917cff9640 10 monclient(hunting): dump: 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:11.537 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:11.538 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:11.538 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:11.538 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:11.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2374238962 0 0) 0x7f9180141ec0 con 0x7f918014be90 2024-03-25T19:45:11.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 >> v1:172.21.15.195:6789/0 conn(0x7f91800b4d60 legacy=0x7f91801d7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 >> v1:172.21.15.107:6789/0 conn(0x7f91800b1730 legacy=0x7f9180140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f917cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:45:11.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f917cff9640 1 monclient: found mon.c 2024-03-25T19:45:11.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f917cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:11.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f917cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:11.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91801d8590 con 0x7f918014be90 2024-03-25T19:45:11.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f917cff9640 10 monclient: _finish_auth 0 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f917cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f917cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f9186939640 5 monclient: authenticate success, global_id 4649 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f9186939640 10 monclient: _renew_subs 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f9186939640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f9186939640 1 -- 172.21.15.107:0/1560351512 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9180142530 con 0x7f918014be90 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f9186939640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f9186939640 10 monclient: _renew_subs 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f9186939640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f9186939640 1 -- 172.21.15.107:0/1560351512 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f918013b790 con 0x7f918014be90 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.545+0000 7f9186939640 1 librados: init done 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.546+0000 7f9186939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.546+0000 7f9186939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.546+0000 7f9186939640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:11.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.546+0000 7f9186939640 1 -- 172.21.15.107:0/1560351512 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91801d8780 con 0x7f918014be90 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.908+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91780020e0 con 0x7f918014be90 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.908+0000 7f917cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.908+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f9178003990 con 0x7f918014be90 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.908+0000 7f917cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.908+0000 7f917cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.908+0000 7f917cff9640 10 monclient: dump: 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:11.908 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:11.909 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:11.909 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:11.909 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:11.909 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:11.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.908+0000 7f917e7fc640 4 set_mon_vals no callback set 2024-03-25T19:45:11.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.908+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f9178003bb0 con 0x7f918014be90 2024-03-25T19:45:11.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.909+0000 7f917ffff640 10 monclient: get_auth_request con 0x7f916c06dfd0 auth_method 0 2024-03-25T19:45:11.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.909+0000 7f917cff9640 10 client.4649.objecter ms_handle_connect 0x7f916c06dfd0 2024-03-25T19:45:11.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.915+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f91780547b0 con 0x7f918014be90 2024-03-25T19:45:11.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.915+0000 7f917cff9640 10 client.4649.objecter ms_dispatch 0x7f9180000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:11.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.915+0000 7f917cff9640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:11.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.915+0000 7f917cff9640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:11.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.915+0000 7f917cff9640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-25T19:45:11.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.915+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91780500f0 con 0x7f918014be90 2024-03-25T19:45:11.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.915+0000 7f917cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:11.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:11.915+0000 7f917cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:12.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.179+0000 7f9186939640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:12.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.179+0000 7f9186939640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.179+0000 7f9186939640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.179+0000 7f9186939640 1 -- 172.21.15.107:0/1560351512 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91800afbd0 con 0x7f918014be90 2024-03-25T19:45:12.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.208+0000 7f917cff9640 1 -- 172.21.15.107:0/1560351512 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f91780521f0 con 0x7f918014be90 2024-03-25T19:45:12.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.208+0000 7f917cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:12.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.208+0000 7f917cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:12.208 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:45:12.208 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:12.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 10 librados: watch_flush enter 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 10 librados: watch_flush exit 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 10 client.4649.objecter successfully canceled tick 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 1 -- 172.21.15.107:0/1560351512 >> v1:172.21.15.195:6800/2984408421 conn(0x7f916c06dfd0 legacy=0x7f916c070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 10 monclient: shutdown 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 1 -- 172.21.15.107:0/1560351512 >> v1:172.21.15.107:6790/0 conn(0x7f918014be90 legacy=0x7f9180142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 1 -- 172.21.15.107:0/1560351512 shutdown_connections 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 1 -- 172.21.15.107:0/1560351512 >> 172.21.15.107:0/1560351512 conn(0x7f91800617a0 msgr2=0x7f91800b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 1 -- 172.21.15.107:0/1560351512 shutdown_connections 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.210+0000 7f9186939640 1 -- 172.21.15.107:0/1560351512 wait complete. 2024-03-25T19:45:12.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:12.211+0000 7f9186939640 1 librados: shutdown 2024-03-25T19:45:12.223 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:19.029 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:19.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.147+0000 7f904996b640 10 monclient: get_monmap_and_config 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.147+0000 7f904996b640 10 monclient: build_initial_monmap 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.147+0000 7f904996b640 10 monclient: monmap: 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:45:19.148790+0000 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:45:19.148790+0000 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:45:19.148 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:19.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.148+0000 7f904996b640 1 Processor -- start 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.148+0000 7f904996b640 1 -- start start 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.148+0000 7f904996b640 10 monclient: init 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.149+0000 7f904996b640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.149+0000 7f904996b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.149+0000 7f904996b640 10 monclient(hunting): picked mon.noname-b con 0x7f90440a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.149+0000 7f904996b640 10 monclient(hunting): picked mon.noname-a con 0x7f904413f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.149+0000 7f904996b640 10 monclient(hunting): picked mon.noname-c con 0x7f90441430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.149+0000 7f904996b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9044142520 con 0x7f904413f710 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.149+0000 7f904996b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90440b28a0 con 0x7f90441430c0 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.149+0000 7f904996b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90440b2a30 con 0x7f90440a98f0 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.149+0000 7f904996b640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.149+0000 7f904996b640 10 monclient(hunting): authenticate will time out at 5111.490482s 2024-03-25T19:45:19.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.158+0000 7f903bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f904413f710 0x7f90441429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34586/0 (socket says 172.21.15.107:34586) 2024-03-25T19:45:19.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.158+0000 7f903bfff640 1 -- 172.21.15.107:0/3877081942 learned_addr learned my addr 172.21.15.107:0/3877081942 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.169+0000 7f90427fc640 1 -- 172.21.15.107:0/3877081942 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014318722 0 0) 0x7f9044142520 con 0x7f904413f710 2024-03-25T19:45:19.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.169+0000 7f90427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:19.169 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.169+0000 7f90427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:19.169 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.169+0000 7f90427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:19.169 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.169+0000 7f90427fc640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:45:19.169 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.169+0000 7f90427fc640 1 -- 172.21.15.107:0/3877081942 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9030003520 con 0x7f904413f710 2024-03-25T19:45:19.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.177+0000 7f90427fc640 1 -- 172.21.15.107:0/3877081942 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783464527 0 0) 0x7f90440b28a0 con 0x7f90441430c0 2024-03-25T19:45:19.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.177+0000 7f90427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:19.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.177+0000 7f90427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:19.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.177+0000 7f90427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:19.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.177+0000 7f90427fc640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:45:19.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.177+0000 7f90427fc640 1 -- 172.21.15.107:0/3877081942 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9044142520 con 0x7f90441430c0 2024-03-25T19:45:19.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 1 -- 172.21.15.107:0/3877081942 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2298765484 0 0) 0x7f9030003520 con 0x7f904413f710 2024-03-25T19:45:19.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 1 -- 172.21.15.107:0/3877081942 >> v1:172.21.15.107:6790/0 conn(0x7f90440a98f0 legacy=0x7f90440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:19.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 1 -- 172.21.15.107:0/3877081942 >> v1:172.21.15.195:6789/0 conn(0x7f90441430c0 legacy=0x7f904414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:19.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 1 monclient: found mon.noname-a 2024-03-25T19:45:19.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:19.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:45:19.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 1 -- 172.21.15.107:0/3877081942 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90440b44c0 con 0x7f904413f710 2024-03-25T19:45:19.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 1 -- 172.21.15.107:0/3877081942 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f902c001690 con 0x7f904413f710 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f904996b640 5 monclient: authenticate success, global_id 4647 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f904996b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f90427fc640 10 monclient: dump: 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:19.184 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:19.185 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:19.185 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:19.185 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.183+0000 7f904996b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:19.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.193+0000 7f90427fc640 1 -- 172.21.15.107:0/3877081942 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f902c0017f0 con 0x7f904413f710 2024-03-25T19:45:19.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.193+0000 7f90427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:19.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.193+0000 7f90427fc640 1 -- 172.21.15.107:0/3877081942 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f902c001c00 con 0x7f904413f710 2024-03-25T19:45:19.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.193+0000 7f90427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:19.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.193+0000 7f90427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:19.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.193+0000 7f90427fc640 10 monclient: dump: 2024-03-25T19:45:19.193 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:19.194 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:19.194 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:19.194 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:19.194 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:19.194 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:19.194 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:19.194 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:19.194 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:19.194 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:19.194 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.193+0000 7f904996b640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:19.194 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.193+0000 7f904996b640 4 set_mon_vals no callback set 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.194+0000 7f904996b640 10 monclient: shutdown 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.194+0000 7f904996b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.194+0000 7f904996b640 1 -- 172.21.15.107:0/3877081942 >> v1:172.21.15.107:6789/0 conn(0x7f904413f710 legacy=0x7f90441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.194+0000 7f904996b640 1 -- 172.21.15.107:0/3877081942 shutdown_connections 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.194+0000 7f904996b640 1 -- 172.21.15.107:0/3877081942 >> 172.21.15.107:0/3877081942 conn(0x7f90440b7ea0 msgr2=0x7f90440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.194+0000 7f904996b640 1 -- 172.21.15.107:0/3877081942 shutdown_connections 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.194+0000 7f904996b640 1 -- 172.21.15.107:0/3877081942 wait complete. 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.195+0000 7f904996b640 10 monclient: build_initial_monmap 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.195+0000 7f904996b640 10 monclient: monmap: 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:45:19.196 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.195+0000 7f904996b640 1 librados: starting msgr at 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.195+0000 7f904996b640 1 librados: starting objecter 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 1 Processor -- start 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 1 -- start start 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 1 librados: setting wanted keys 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 1 librados: calling monclient init 2024-03-25T19:45:19.197 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 10 monclient: init 2024-03-25T19:45:19.198 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:19.198 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:19.198 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 10 monclient(hunting): picked mon.noname-b con 0x7f90440a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:19.198 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 10 monclient(hunting): picked mon.noname-c con 0x7f904413f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:19.198 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 10 monclient(hunting): picked mon.noname-a con 0x7f90441430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:19.198 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904414d280 con 0x7f90441430c0 2024-03-25T19:45:19.198 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904414d4f0 con 0x7f90440a98f0 2024-03-25T19:45:19.198 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904414d7d0 con 0x7f904413f710 2024-03-25T19:45:19.199 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:19.199 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.196+0000 7f904996b640 10 monclient(hunting): authenticate will time out at 5111.537694s 2024-03-25T19:45:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.208+0000 7f9042ffd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f90440a98f0 0x7f904414bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:39948/0 (socket says 172.21.15.107:39948) 2024-03-25T19:45:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.208+0000 7f9042ffd640 1 -- 172.21.15.107:0/2969928265 learned_addr learned my addr 172.21.15.107:0/2969928265 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.208+0000 7f9040ff9640 10 client.?.objecter ms_handle_connect 0x7f90440a98f0 2024-03-25T19:45:19.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.208+0000 7f9040ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:19.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.212+0000 7f9040ff9640 10 client.?.objecter ms_handle_connect 0x7f904413f710 2024-03-25T19:45:19.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.212+0000 7f9040ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:19.215 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.216+0000 7f9040ff9640 10 client.?.objecter ms_handle_connect 0x7f90441430c0 2024-03-25T19:45:19.215 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.216+0000 7f9040ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:19.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.229+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523543672 0 0) 0x7f904414d7d0 con 0x7f904413f710 2024-03-25T19:45:19.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.229+0000 7f9040ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:19.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.229+0000 7f9040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:19.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.229+0000 7f9040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:19.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.229+0000 7f9040ff9640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:45:19.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.229+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9024004eb0 con 0x7f904413f710 2024-03-25T19:45:19.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.284+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615907488 0 0) 0x7f904414d4f0 con 0x7f90440a98f0 2024-03-25T19:45:19.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.284+0000 7f9040ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:19.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.284+0000 7f9040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:19.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.284+0000 7f9040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:19.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.284+0000 7f9040ff9640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:45:19.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.284+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904414d7d0 con 0x7f90440a98f0 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.298+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1629648934 0 0) 0x7f9024004eb0 con 0x7f904413f710 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.298+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f904414d4f0 con 0x7f904413f710 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.298+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f902c001b00 con 0x7f904413f710 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.298+0000 7f9040ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.298+0000 7f9040ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.298+0000 7f9040ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:19.298 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:19.299 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:19.299 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:19.299 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:19.299 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:19.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.298+0000 7f9040ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:19.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.308+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 55117454 0 0) 0x7f904414d7d0 con 0x7f90440a98f0 2024-03-25T19:45:19.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.309+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9024004eb0 con 0x7f90440a98f0 2024-03-25T19:45:19.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.309+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f90340018e0 con 0x7f90440a98f0 2024-03-25T19:45:19.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.309+0000 7f9040ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.309+0000 7f9040ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.309+0000 7f9040ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:19.309 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:19.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.310+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046913609 0 0) 0x7f904414d280 con 0x7f90441430c0 2024-03-25T19:45:19.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.310+0000 7f9040ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:19.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.310+0000 7f9040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:19.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.310+0000 7f9040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:19.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.310+0000 7f9040ff9640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:45:19.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.310+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904414d7d0 con 0x7f90441430c0 2024-03-25T19:45:19.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 449970858 0 0) 0x7f904414d4f0 con 0x7f904413f710 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 >> v1:172.21.15.195:6789/0 conn(0x7f90440a98f0 legacy=0x7f904414bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 >> v1:172.21.15.107:6789/0 conn(0x7f90441430c0 legacy=0x7f90441e72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f9040ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f9040ff9640 1 monclient: found mon.c 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f9040ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f9040ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904412dcf0 con 0x7f904413f710 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f9040ff9640 10 monclient: _finish_auth 0 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f9040ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f9040ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f904996b640 5 monclient: authenticate success, global_id 4652 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f904996b640 10 monclient: _renew_subs 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f904996b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:19.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.322+0000 7f904996b640 1 -- 172.21.15.107:0/2969928265 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90440b7040 con 0x7f904413f710 2024-03-25T19:45:19.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.323+0000 7f904996b640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:19.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.323+0000 7f904996b640 10 monclient: _renew_subs 2024-03-25T19:45:19.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.323+0000 7f904996b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:19.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.323+0000 7f904996b640 1 -- 172.21.15.107:0/2969928265 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9044139850 con 0x7f904413f710 2024-03-25T19:45:19.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.323+0000 7f904996b640 1 librados: init done 2024-03-25T19:45:19.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.324+0000 7f904996b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:19.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.324+0000 7f904996b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:19.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.324+0000 7f904996b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:19.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.324+0000 7f904996b640 1 -- 172.21.15.107:0/2969928265 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90440627b0 con 0x7f904413f710 2024-03-25T19:45:19.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.625+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f902c004a90 con 0x7f904413f710 2024-03-25T19:45:19.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.625+0000 7f9040ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:19.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.625+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f902c004da0 con 0x7f904413f710 2024-03-25T19:45:19.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.625+0000 7f9040ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:19.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.625+0000 7f9040ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:19.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.625+0000 7f9040ff9640 10 monclient: dump: 2024-03-25T19:45:19.625 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:19.626 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:19.626 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:19.626 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:19.626 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:19.626 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:19.626 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:19.626 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:19.626 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:19.626 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:19.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.625+0000 7f9041ffb640 4 set_mon_vals no callback set 2024-03-25T19:45:19.627 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.626+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f902c01ad70 con 0x7f904413f710 2024-03-25T19:45:19.627 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.626+0000 7f9042ffd640 10 monclient: get_auth_request con 0x7f902406df50 auth_method 0 2024-03-25T19:45:19.627 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.627+0000 7f9040ff9640 10 client.4652.objecter ms_handle_connect 0x7f902406df50 2024-03-25T19:45:19.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.633+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f902c054870 con 0x7f904413f710 2024-03-25T19:45:19.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.633+0000 7f9040ff9640 10 client.4652.objecter ms_dispatch 0x7f9044000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:19.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.633+0000 7f9040ff9640 3 client.4652.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:19.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.633+0000 7f9040ff9640 3 client.4652.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:19.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.633+0000 7f9040ff9640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-25T19:45:19.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.633+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f902c050380 con 0x7f904413f710 2024-03-25T19:45:19.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.633+0000 7f9040ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:19.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.633+0000 7f9040ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:19.879 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.879+0000 7f904996b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:19.879 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.879+0000 7f904996b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:19.879 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.879+0000 7f904996b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:19.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.879+0000 7f904996b640 1 -- 172.21.15.107:0/2969928265 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9044074cc0 con 0x7f904413f710 2024-03-25T19:45:19.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.908+0000 7f9040ff9640 1 -- 172.21.15.107:0/2969928265 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f902c0522b0 con 0x7f904413f710 2024-03-25T19:45:19.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.908+0000 7f9040ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:19.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.908+0000 7f9040ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:19.909 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:45:19.909 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:19.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 10 librados: watch_flush enter 2024-03-25T19:45:19.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 10 librados: watch_flush exit 2024-03-25T19:45:19.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-25T19:45:19.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 10 client.4652.objecter successfully canceled tick 2024-03-25T19:45:19.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 1 -- 172.21.15.107:0/2969928265 >> v1:172.21.15.195:6800/2984408421 conn(0x7f902406df50 legacy=0x7f9024070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 10 monclient: shutdown 2024-03-25T19:45:19.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:19.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 1 -- 172.21.15.107:0/2969928265 >> v1:172.21.15.107:6790/0 conn(0x7f904413f710 legacy=0x7f9044139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:19.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 1 -- 172.21.15.107:0/2969928265 shutdown_connections 2024-03-25T19:45:19.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 1 -- 172.21.15.107:0/2969928265 >> 172.21.15.107:0/2969928265 conn(0x7f90440b7ea0 msgr2=0x7f9044074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:19.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 1 -- 172.21.15.107:0/2969928265 shutdown_connections 2024-03-25T19:45:19.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 1 -- 172.21.15.107:0/2969928265 wait complete. 2024-03-25T19:45:19.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:19.911+0000 7f904996b640 1 librados: shutdown 2024-03-25T19:45:19.924 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:26.731 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:26.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.848+0000 7f3f41663640 10 monclient: get_monmap_and_config 2024-03-25T19:45:26.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.848+0000 7f3f41663640 10 monclient: build_initial_monmap 2024-03-25T19:45:26.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.849+0000 7f3f41663640 10 monclient: monmap: 2024-03-25T19:45:26.850 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:26.850 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:26.850 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:45:26.850975+0000 2024-03-25T19:45:26.850 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:45:26.850975+0000 2024-03-25T19:45:26.850 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:26.850 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:26.850 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:26.850 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:45:26.850 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:45:26.850 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.849+0000 7f3f41663640 1 Processor -- start 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.849+0000 7f3f41663640 1 -- start start 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.849+0000 7f3f41663640 10 monclient: init 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.850+0000 7f3f41663640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.850+0000 7f3f41663640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.850+0000 7f3f41663640 10 monclient(hunting): picked mon.noname-b con 0x7f3f3c0a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.850+0000 7f3f41663640 10 monclient(hunting): picked mon.noname-a con 0x7f3f3c13f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.850+0000 7f3f41663640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3c1430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.850+0000 7f3f41663640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c142520 con 0x7f3f3c13f710 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.850+0000 7f3f41663640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0b28a0 con 0x7f3f3c1430c0 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.850+0000 7f3f41663640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0b2a30 con 0x7f3f3c0a98f0 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.850+0000 7f3f41663640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.850+0000 7f3f41663640 10 monclient(hunting): authenticate will time out at 5119.192315s 2024-03-25T19:45:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.857+0000 7f3f3b7fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f3f3c1430c0 0x7f3f3c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:55696/0 (socket says 172.21.15.107:55696) 2024-03-25T19:45:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.857+0000 7f3f3b7fe640 1 -- 172.21.15.107:0/2578628463 learned_addr learned my addr 172.21.15.107:0/2578628463 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:26.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.873+0000 7f3f39ffb640 1 -- 172.21.15.107:0/2578628463 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316884554 0 0) 0x7f3f3c0b28a0 con 0x7f3f3c1430c0 2024-03-25T19:45:26.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.873+0000 7f3f39ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:26.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.873+0000 7f3f39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:26.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.873+0000 7f3f39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:26.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.873+0000 7f3f39ffb640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:45:26.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.873+0000 7f3f39ffb640 1 -- 172.21.15.107:0/2578628463 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f24003520 con 0x7f3f3c1430c0 2024-03-25T19:45:26.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.882+0000 7f3f39ffb640 1 -- 172.21.15.107:0/2578628463 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741782596 0 0) 0x7f3f3c0b2a30 con 0x7f3f3c0a98f0 2024-03-25T19:45:26.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.882+0000 7f3f39ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:26.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.882+0000 7f3f39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:26.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.882+0000 7f3f39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:26.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.882+0000 7f3f39ffb640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:45:26.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.882+0000 7f3f39ffb640 1 -- 172.21.15.107:0/2578628463 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f3c0b28a0 con 0x7f3f3c0a98f0 2024-03-25T19:45:26.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 1 -- 172.21.15.107:0/2578628463 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157935346 0 0) 0x7f3f24003520 con 0x7f3f3c1430c0 2024-03-25T19:45:26.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 1 -- 172.21.15.107:0/2578628463 >> v1:172.21.15.107:6790/0 conn(0x7f3f3c0a98f0 legacy=0x7f3f3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 1 -- 172.21.15.107:0/2578628463 >> v1:172.21.15.107:6789/0 conn(0x7f3f3c13f710 legacy=0x7f3f3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:26.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 1 monclient: found mon.noname-c 2024-03-25T19:45:26.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:26.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 1 -- 172.21.15.107:0/2578628463 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0b44c0 con 0x7f3f3c1430c0 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 1 -- 172.21.15.107:0/2578628463 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3f34001a10 con 0x7f3f3c1430c0 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f39ffb640 10 monclient: dump: 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:26.887 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:26.888 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:26.888 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:26.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f41663640 5 monclient: authenticate success, global_id 4645 2024-03-25T19:45:26.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.885+0000 7f3f41663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:26.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.892+0000 7f3f39ffb640 1 -- 172.21.15.107:0/2578628463 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f34001b70 con 0x7f3f3c1430c0 2024-03-25T19:45:26.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.892+0000 7f3f39ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:26.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.892+0000 7f3f39ffb640 1 -- 172.21.15.107:0/2578628463 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3f34001f80 con 0x7f3f3c1430c0 2024-03-25T19:45:26.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.892+0000 7f3f41663640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:26.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.892+0000 7f3f41663640 4 set_mon_vals no callback set 2024-03-25T19:45:26.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.892+0000 7f3f39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:26.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.892+0000 7f3f39ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:26.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.892+0000 7f3f39ffb640 10 monclient: dump: 2024-03-25T19:45:26.893 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:26.893 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:26.893 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:26.894 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:26.894 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:26.894 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:26.894 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:26.894 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:26.894 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:26.894 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:26.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.893+0000 7f3f41663640 10 monclient: shutdown 2024-03-25T19:45:26.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.893+0000 7f3f41663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:26.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.893+0000 7f3f41663640 1 -- 172.21.15.107:0/2578628463 >> v1:172.21.15.195:6789/0 conn(0x7f3f3c1430c0 legacy=0x7f3f3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.893+0000 7f3f41663640 1 -- 172.21.15.107:0/2578628463 shutdown_connections 2024-03-25T19:45:26.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.893+0000 7f3f41663640 1 -- 172.21.15.107:0/2578628463 >> 172.21.15.107:0/2578628463 conn(0x7f3f3c0b7ea0 msgr2=0x7f3f3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:26.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.893+0000 7f3f41663640 1 -- 172.21.15.107:0/2578628463 shutdown_connections 2024-03-25T19:45:26.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.893+0000 7f3f41663640 1 -- 172.21.15.107:0/2578628463 wait complete. 2024-03-25T19:45:26.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.894+0000 7f3f41663640 10 monclient: build_initial_monmap 2024-03-25T19:45:26.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.894+0000 7f3f41663640 10 monclient: monmap: 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.894+0000 7f3f41663640 1 librados: starting msgr at 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.894+0000 7f3f41663640 1 librados: starting objecter 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.894+0000 7f3f41663640 1 Processor -- start 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.894+0000 7f3f41663640 1 -- start start 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.894+0000 7f3f41663640 1 librados: setting wanted keys 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.894+0000 7f3f41663640 1 librados: calling monclient init 2024-03-25T19:45:26.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.894+0000 7f3f41663640 10 monclient: init 2024-03-25T19:45:26.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.895+0000 7f3f41663640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:26.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.895+0000 7f3f41663640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:26.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.895+0000 7f3f41663640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3c0a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:26.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.895+0000 7f3f41663640 10 monclient(hunting): picked mon.noname-b con 0x7f3f3c13f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:26.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.895+0000 7f3f41663640 10 monclient(hunting): picked mon.noname-a con 0x7f3f3c1430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:26.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.895+0000 7f3f41663640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0abd30 con 0x7f3f3c1430c0 2024-03-25T19:45:26.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.895+0000 7f3f41663640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c1d7810 con 0x7f3f3c13f710 2024-03-25T19:45:26.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.895+0000 7f3f41663640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c1d8440 con 0x7f3f3c0a98f0 2024-03-25T19:45:26.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.895+0000 7f3f41663640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:26.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.895+0000 7f3f41663640 10 monclient(hunting): authenticate will time out at 5119.237279s 2024-03-25T19:45:26.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.899+0000 7f3f3a7fc640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f3f3c13f710 0x7f3f3c1d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:55712/0 (socket says 172.21.15.107:55712) 2024-03-25T19:45:26.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.899+0000 7f3f3a7fc640 1 -- 172.21.15.107:0/2732283146 learned_addr learned my addr 172.21.15.107:0/2732283146 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:26.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.899+0000 7f3f237fe640 10 client.?.objecter ms_handle_connect 0x7f3f3c13f710 2024-03-25T19:45:26.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.899+0000 7f3f237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:26.903 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.903+0000 7f3f237fe640 10 client.?.objecter ms_handle_connect 0x7f3f3c0a98f0 2024-03-25T19:45:26.903 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.903+0000 7f3f237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:26.905 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.904+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193914880 0 0) 0x7f3f3c1d7810 con 0x7f3f3c13f710 2024-03-25T19:45:26.905 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.904+0000 7f3f237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:26.905 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.904+0000 7f3f237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:26.905 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.904+0000 7f3f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:26.905 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.904+0000 7f3f237fe640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:45:26.905 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.904+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f28004eb0 con 0x7f3f3c13f710 2024-03-25T19:45:26.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.914+0000 7f3f237fe640 10 client.?.objecter ms_handle_connect 0x7f3f3c1430c0 2024-03-25T19:45:26.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.914+0000 7f3f237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:26.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.916+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310858703 0 0) 0x7f3f3c1d8440 con 0x7f3f3c0a98f0 2024-03-25T19:45:26.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.916+0000 7f3f237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:26.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.916+0000 7f3f237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:26.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.916+0000 7f3f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:26.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.916+0000 7f3f237fe640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:45:26.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.916+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f3c1d7810 con 0x7f3f3c0a98f0 2024-03-25T19:45:26.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.917+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3334839966 0 0) 0x7f3f28004eb0 con 0x7f3f3c13f710 2024-03-25T19:45:26.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.917+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f3c1d8440 con 0x7f3f3c13f710 2024-03-25T19:45:26.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.917+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3f300019d0 con 0x7f3f3c13f710 2024-03-25T19:45:26.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.917+0000 7f3f237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:26.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.917+0000 7f3f237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:26.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.917+0000 7f3f237fe640 10 monclient(hunting): dump: 2024-03-25T19:45:26.918 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:26.918 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:26.918 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:26.919 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:26.919 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:26.919 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:26.919 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:26.919 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:26.919 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:26.919 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:26.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.917+0000 7f3f237fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:26.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491333887 0 0) 0x7f3f3c0abd30 con 0x7f3f3c1430c0 2024-03-25T19:45:26.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:26.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:26.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:26.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:45:26.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f28004eb0 con 0x7f3f3c1430c0 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 614382394 0 0) 0x7f3f3c1d8440 con 0x7f3f3c13f710 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 >> v1:172.21.15.107:6790/0 conn(0x7f3f3c0a98f0 legacy=0x7f3f3c1d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 >> v1:172.21.15.107:6789/0 conn(0x7f3f3c1430c0 legacy=0x7f3f3c1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 1 monclient: found mon.b 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c1d5000 con 0x7f3f3c13f710 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f41663640 5 monclient: authenticate success, global_id 4648 2024-03-25T19:45:26.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f41663640 10 monclient: _renew_subs 2024-03-25T19:45:26.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f41663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:45:26.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.923+0000 7f3f41663640 1 -- 172.21.15.107:0/2732283146 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c1d5250 con 0x7f3f3c13f710 2024-03-25T19:45:26.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.924+0000 7f3f41663640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:26.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.924+0000 7f3f41663640 10 monclient: _renew_subs 2024-03-25T19:45:26.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.924+0000 7f3f41663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:45:26.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.924+0000 7f3f41663640 1 -- 172.21.15.107:0/2732283146 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c1d43c0 con 0x7f3f3c13f710 2024-03-25T19:45:26.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.924+0000 7f3f41663640 1 librados: init done 2024-03-25T19:45:26.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.925+0000 7f3f41663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:26.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.925+0000 7f3f41663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:26.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.925+0000 7f3f41663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:45:26.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:26.925+0000 7f3f41663640 1 -- 172.21.15.107:0/2732283146 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c1d7670 con 0x7f3f3c13f710 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.229+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f30003610 con 0x7f3f3c13f710 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.229+0000 7f3f237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.229+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f3f30003800 con 0x7f3f3c13f710 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.229+0000 7f3f237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.229+0000 7f3f237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.229+0000 7f3f237fe640 10 monclient: dump: 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:27.230 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:27.231 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:27.231 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:27.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.229+0000 7f3f397fa640 4 set_mon_vals no callback set 2024-03-25T19:45:27.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.230+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f3f30003a20 con 0x7f3f3c13f710 2024-03-25T19:45:27.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.231+0000 7f3f3affd640 10 monclient: get_auth_request con 0x7f3f2806de00 auth_method 0 2024-03-25T19:45:27.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.231+0000 7f3f237fe640 10 client.4648.objecter ms_handle_connect 0x7f3f2806de00 2024-03-25T19:45:27.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.234+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f3f30054650 con 0x7f3f3c13f710 2024-03-25T19:45:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.234+0000 7f3f237fe640 10 client.4648.objecter ms_dispatch 0x7f3f3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.234+0000 7f3f237fe640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.234+0000 7f3f237fe640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.234+0000 7f3f237fe640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T19:45:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.234+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f30050160 con 0x7f3f3c13f710 2024-03-25T19:45:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.234+0000 7f3f237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.234+0000 7f3f237fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:27.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.463+0000 7f3f41663640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:27.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.463+0000 7f3f41663640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:27.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.463+0000 7f3f41663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:45:27.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.463+0000 7f3f41663640 1 -- 172.21.15.107:0/2732283146 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f3c0abb70 con 0x7f3f3c13f710 2024-03-25T19:45:27.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.497+0000 7f3f237fe640 1 -- 172.21.15.107:0/2732283146 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f3f30052090 con 0x7f3f3c13f710 2024-03-25T19:45:27.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.497+0000 7f3f237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:27.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.497+0000 7f3f237fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:27.498 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:45:27.498 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.499+0000 7f3f41663640 10 librados: watch_flush enter 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.499+0000 7f3f41663640 10 librados: watch_flush exit 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.499+0000 7f3f41663640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.499+0000 7f3f41663640 10 client.4648.objecter successfully canceled tick 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.500+0000 7f3f41663640 1 -- 172.21.15.107:0/2732283146 >> v1:172.21.15.195:6800/2984408421 conn(0x7f3f2806de00 legacy=0x7f3f280702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.500+0000 7f3f41663640 10 monclient: shutdown 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.500+0000 7f3f41663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.500+0000 7f3f41663640 1 -- 172.21.15.107:0/2732283146 >> v1:172.21.15.195:6789/0 conn(0x7f3f3c13f710 legacy=0x7f3f3c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.500+0000 7f3f41663640 1 -- 172.21.15.107:0/2732283146 shutdown_connections 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.500+0000 7f3f41663640 1 -- 172.21.15.107:0/2732283146 >> 172.21.15.107:0/2732283146 conn(0x7f3f3c0b7ea0 msgr2=0x7f3f3c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.500+0000 7f3f41663640 1 -- 172.21.15.107:0/2732283146 shutdown_connections 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.500+0000 7f3f41663640 1 -- 172.21.15.107:0/2732283146 wait complete. 2024-03-25T19:45:27.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:27.500+0000 7f3f41663640 1 librados: shutdown 2024-03-25T19:45:27.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:31.332 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:10:01.648 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:31.390 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:10:01.706 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:31.430 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:10:01.746 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:31.438 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:10:01.724 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:31.444 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:10:01.760 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:31.444 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:10:01.731 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:31.450 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:10:01.766 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:31.468 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:10:01.755 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:31.477 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:10:01.793 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:34.319 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:34.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.435+0000 7fba91257640 10 monclient: get_monmap_and_config 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.435+0000 7fba91257640 10 monclient: build_initial_monmap 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.435+0000 7fba91257640 10 monclient: monmap: 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:45:34.436129+0000 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:45:34.436129+0000 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:45:34.435 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.435+0000 7fba91257640 1 Processor -- start 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.435+0000 7fba91257640 1 -- start start 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.435+0000 7fba91257640 10 monclient: init 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.436+0000 7fba91257640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.436+0000 7fba91257640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.436+0000 7fba91257640 10 monclient(hunting): picked mon.noname-c con 0x7fba8c0a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.436+0000 7fba91257640 10 monclient(hunting): picked mon.noname-a con 0x7fba8c13f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.436+0000 7fba91257640 10 monclient(hunting): picked mon.noname-b con 0x7fba8c1430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.436+0000 7fba91257640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8c142520 con 0x7fba8c13f710 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.436+0000 7fba91257640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8c0b28a0 con 0x7fba8c0a98f0 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.436+0000 7fba91257640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8c0b2a30 con 0x7fba8c1430c0 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.436+0000 7fba91257640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:34.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.436+0000 7fba91257640 10 monclient(hunting): authenticate will time out at 5126.777415s 2024-03-25T19:45:34.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.445+0000 7fba8ad9d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fba8c0a98f0 0x7fba8c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:55524/0 (socket says 172.21.15.107:55524) 2024-03-25T19:45:34.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.445+0000 7fba8ad9d640 1 -- 172.21.15.107:0/2100776511 learned_addr learned my addr 172.21.15.107:0/2100776511 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:34.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.510+0000 7fba89d9b640 1 -- 172.21.15.107:0/2100776511 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528305861 0 0) 0x7fba8c0b2a30 con 0x7fba8c1430c0 2024-03-25T19:45:34.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.510+0000 7fba89d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:34.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.510+0000 7fba89d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:34.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.510+0000 7fba89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:34.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.510+0000 7fba89d9b640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:45:34.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.510+0000 7fba89d9b640 1 -- 172.21.15.107:0/2100776511 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba74003520 con 0x7fba8c1430c0 2024-03-25T19:45:34.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 1 -- 172.21.15.107:0/2100776511 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 458089328 0 0) 0x7fba74003520 con 0x7fba8c1430c0 2024-03-25T19:45:34.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 1 -- 172.21.15.107:0/2100776511 >> v1:172.21.15.195:6789/0 conn(0x7fba8c0a98f0 legacy=0x7fba8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 1 -- 172.21.15.107:0/2100776511 >> v1:172.21.15.107:6789/0 conn(0x7fba8c13f710 legacy=0x7fba8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:45:34.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 1 monclient: found mon.noname-b 2024-03-25T19:45:34.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:34.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:45:34.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 1 -- 172.21.15.107:0/2100776511 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c0b44c0 con 0x7fba8c1430c0 2024-03-25T19:45:34.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 10 monclient: _finish_auth 0 2024-03-25T19:45:34.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 1 -- 172.21.15.107:0/2100776511 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fba84001a10 con 0x7fba8c1430c0 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba89d9b640 10 monclient: dump: 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:34.539 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:34.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba91257640 5 monclient: authenticate success, global_id 4661 2024-03-25T19:45:34.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.538+0000 7fba91257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:34.550 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.551+0000 7fba89d9b640 1 -- 172.21.15.107:0/2100776511 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba84001b70 con 0x7fba8c1430c0 2024-03-25T19:45:34.550 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.551+0000 7fba89d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.551+0000 7fba89d9b640 1 -- 172.21.15.107:0/2100776511 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fba84001f80 con 0x7fba8c1430c0 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.551+0000 7fba89d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.551+0000 7fba89d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.551+0000 7fba89d9b640 10 monclient: dump: 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.551+0000 7fba91257640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:34.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.551+0000 7fba91257640 4 set_mon_vals no callback set 2024-03-25T19:45:34.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.551+0000 7fba91257640 10 monclient: shutdown 2024-03-25T19:45:34.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.551+0000 7fba91257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:34.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.551+0000 7fba91257640 1 -- 172.21.15.107:0/2100776511 >> v1:172.21.15.107:6790/0 conn(0x7fba8c1430c0 legacy=0x7fba8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.552+0000 7fba91257640 1 -- 172.21.15.107:0/2100776511 shutdown_connections 2024-03-25T19:45:34.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.552+0000 7fba91257640 1 -- 172.21.15.107:0/2100776511 >> 172.21.15.107:0/2100776511 conn(0x7fba8c0b7ea0 msgr2=0x7fba8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:34.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.552+0000 7fba91257640 1 -- 172.21.15.107:0/2100776511 shutdown_connections 2024-03-25T19:45:34.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.552+0000 7fba91257640 1 -- 172.21.15.107:0/2100776511 wait complete. 2024-03-25T19:45:34.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.553+0000 7fba91257640 10 monclient: build_initial_monmap 2024-03-25T19:45:34.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.553+0000 7fba91257640 10 monclient: monmap: 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.553+0000 7fba91257640 1 librados: starting msgr at 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.553+0000 7fba91257640 1 librados: starting objecter 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.553+0000 7fba91257640 1 Processor -- start 2024-03-25T19:45:34.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.553+0000 7fba91257640 1 -- start start 2024-03-25T19:45:34.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.553+0000 7fba91257640 1 librados: setting wanted keys 2024-03-25T19:45:34.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.553+0000 7fba91257640 1 librados: calling monclient init 2024-03-25T19:45:34.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.553+0000 7fba91257640 10 monclient: init 2024-03-25T19:45:34.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.554+0000 7fba91257640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:34.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.554+0000 7fba91257640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:34.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.554+0000 7fba91257640 10 monclient(hunting): picked mon.noname-a con 0x7fba8c0a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:34.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.554+0000 7fba91257640 10 monclient(hunting): picked mon.noname-b con 0x7fba8c13f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:34.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.554+0000 7fba91257640 10 monclient(hunting): picked mon.noname-c con 0x7fba8c1430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:34.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.554+0000 7fba91257640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8c136ec0 con 0x7fba8c0a98f0 2024-03-25T19:45:34.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.554+0000 7fba91257640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8c136c20 con 0x7fba8c13f710 2024-03-25T19:45:34.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.554+0000 7fba91257640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8c137db0 con 0x7fba8c1430c0 2024-03-25T19:45:34.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.554+0000 7fba91257640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:34.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.554+0000 7fba91257640 10 monclient(hunting): authenticate will time out at 5126.895284s 2024-03-25T19:45:34.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.561+0000 7fba8b59e640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fba8c1430c0 0x7fba8c1e2c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54612/0 (socket says 172.21.15.107:54612) 2024-03-25T19:45:34.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.561+0000 7fba8b59e640 1 -- 172.21.15.107:0/257339275 learned_addr learned my addr 172.21.15.107:0/257339275 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:34.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.562+0000 7fba737fe640 10 client.?.objecter ms_handle_connect 0x7fba8c1430c0 2024-03-25T19:45:34.561 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.562+0000 7fba737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:34.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.570+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861612511 0 0) 0x7fba8c137db0 con 0x7fba8c1430c0 2024-03-25T19:45:34.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.570+0000 7fba737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:34.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.570+0000 7fba737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:34.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.570+0000 7fba737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:34.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.570+0000 7fba737fe640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:45:34.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.570+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba78004eb0 con 0x7fba8c1430c0 2024-03-25T19:45:34.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.583+0000 7fba737fe640 10 client.?.objecter ms_handle_connect 0x7fba8c0a98f0 2024-03-25T19:45:34.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.583+0000 7fba737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.589+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3533120318 0 0) 0x7fba78004eb0 con 0x7fba8c1430c0 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.589+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba8c137db0 con 0x7fba8c1430c0 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.589+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fba84001fc0 con 0x7fba8c1430c0 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.589+0000 7fba737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.589+0000 7fba737fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.589+0000 7fba737fe640 10 monclient(hunting): dump: 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:34.589 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:34.590 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:34.590 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:34.590 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:34.590 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:34.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.589+0000 7fba737fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:34.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.590+0000 7fba737fe640 10 client.?.objecter ms_handle_connect 0x7fba8c13f710 2024-03-25T19:45:34.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.590+0000 7fba737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:34.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3984628677 0 0) 0x7fba8c137db0 con 0x7fba8c1430c0 2024-03-25T19:45:34.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 >> v1:172.21.15.195:6789/0 conn(0x7fba8c13f710 legacy=0x7fba8c0b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 >> v1:172.21.15.107:6789/0 conn(0x7fba8c0a98f0 legacy=0x7fba8c1376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba737fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:34.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba737fe640 1 monclient: found mon.c 2024-03-25T19:45:34.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:34.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba737fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c138b50 con 0x7fba8c1430c0 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba737fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba737fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba91257640 5 monclient: authenticate success, global_id 4664 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba91257640 10 monclient: _renew_subs 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba91257640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba91257640 1 -- 172.21.15.107:0/257339275 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8c139830 con 0x7fba8c1430c0 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba91257640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba91257640 10 monclient: _renew_subs 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba91257640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba91257640 1 -- 172.21.15.107:0/257339275 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8c1369b0 con 0x7fba8c1430c0 2024-03-25T19:45:34.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.601+0000 7fba91257640 1 librados: init done 2024-03-25T19:45:34.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.602+0000 7fba91257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:34.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.602+0000 7fba91257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:34.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.602+0000 7fba91257640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:34.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.602+0000 7fba91257640 1 -- 172.21.15.107:0/257339275 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8c0627b0 con 0x7fba8c1430c0 2024-03-25T19:45:34.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.969+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba84003c20 con 0x7fba8c1430c0 2024-03-25T19:45:34.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.969+0000 7fba737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:34.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.969+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fba84003ff0 con 0x7fba8c1430c0 2024-03-25T19:45:34.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.969+0000 7fba737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.969+0000 7fba737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.969+0000 7fba737fe640 10 monclient: dump: 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:34.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.969+0000 7fba8959a640 4 set_mon_vals no callback set 2024-03-25T19:45:34.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.970+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fba84004210 con 0x7fba8c1430c0 2024-03-25T19:45:34.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.971+0000 7fba8ad9d640 10 monclient: get_auth_request con 0x7fba7806dc70 auth_method 0 2024-03-25T19:45:34.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.971+0000 7fba737fe640 10 client.4664.objecter ms_handle_connect 0x7fba7806dc70 2024-03-25T19:45:34.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.976+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fba84054c80 con 0x7fba8c1430c0 2024-03-25T19:45:34.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.976+0000 7fba737fe640 10 client.4664.objecter ms_dispatch 0x7fba8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:34.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.976+0000 7fba737fe640 3 client.4664.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:34.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.976+0000 7fba737fe640 3 client.4664.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:34.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.976+0000 7fba737fe640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-25T19:45:34.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.976+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba84050790 con 0x7fba8c1430c0 2024-03-25T19:45:34.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.976+0000 7fba737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:34.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:34.976+0000 7fba737fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:35.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.227+0000 7fba91257640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:35.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.227+0000 7fba91257640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:35.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.227+0000 7fba91257640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:35.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.227+0000 7fba91257640 1 -- 172.21.15.107:0/257339275 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba8c069f40 con 0x7fba8c1430c0 2024-03-25T19:45:35.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.284+0000 7fba737fe640 1 -- 172.21.15.107:0/257339275 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fba840526c0 con 0x7fba8c1430c0 2024-03-25T19:45:35.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.284+0000 7fba737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:35.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.284+0000 7fba737fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:35.284 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:45:35.284 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:35.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.286+0000 7fba91257640 10 librados: watch_flush enter 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.286+0000 7fba91257640 10 librados: watch_flush exit 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.286+0000 7fba91257640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.286+0000 7fba91257640 10 client.4664.objecter successfully canceled tick 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.286+0000 7fba91257640 1 -- 172.21.15.107:0/257339275 >> v1:172.21.15.195:6800/2984408421 conn(0x7fba7806dc70 legacy=0x7fba78070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.286+0000 7fba91257640 10 monclient: shutdown 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.286+0000 7fba91257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.286+0000 7fba91257640 1 -- 172.21.15.107:0/257339275 >> v1:172.21.15.107:6790/0 conn(0x7fba8c1430c0 legacy=0x7fba8c1e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.286+0000 7fba91257640 1 -- 172.21.15.107:0/257339275 shutdown_connections 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.286+0000 7fba91257640 1 -- 172.21.15.107:0/257339275 >> 172.21.15.107:0/257339275 conn(0x7fba8c0b7ea0 msgr2=0x7fba8c13ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.287+0000 7fba91257640 1 -- 172.21.15.107:0/257339275 shutdown_connections 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.287+0000 7fba91257640 1 -- 172.21.15.107:0/257339275 wait complete. 2024-03-25T19:45:35.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:35.287+0000 7fba91257640 1 librados: shutdown 2024-03-25T19:45:35.298 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:42.105 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:42.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.220+0000 7fea009e8640 10 monclient: get_monmap_and_config 2024-03-25T19:45:42.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.220+0000 7fea009e8640 10 monclient: build_initial_monmap 2024-03-25T19:45:42.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.220+0000 7fea009e8640 10 monclient: monmap: 2024-03-25T19:45:42.221 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:42.221 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:42.221 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:45:42.221445+0000 2024-03-25T19:45:42.221 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:45:42.221445+0000 2024-03-25T19:45:42.221 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:42.221 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:42.221 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:42.221 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:45:42.221 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:45:42.222 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:42.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.221+0000 7fea009e8640 1 Processor -- start 2024-03-25T19:45:42.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.221+0000 7fea009e8640 1 -- start start 2024-03-25T19:45:42.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.221+0000 7fea009e8640 10 monclient: init 2024-03-25T19:45:42.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.222+0000 7fea009e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:42.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.222+0000 7fea009e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:42.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.222+0000 7fea009e8640 10 monclient(hunting): picked mon.noname-a con 0x7fe9fc0a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:42.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.222+0000 7fea009e8640 10 monclient(hunting): picked mon.noname-c con 0x7fe9fc13f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:42.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.222+0000 7fea009e8640 10 monclient(hunting): picked mon.noname-b con 0x7fe9fc1430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:42.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.222+0000 7fea009e8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc142520 con 0x7fe9fc0a98f0 2024-03-25T19:45:42.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.222+0000 7fea009e8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc0b28a0 con 0x7fe9fc13f710 2024-03-25T19:45:42.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.222+0000 7fea009e8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc0b2a30 con 0x7fe9fc1430c0 2024-03-25T19:45:42.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.222+0000 7fea009e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:42.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.222+0000 7fea009e8640 10 monclient(hunting): authenticate will time out at 5134.562990s 2024-03-25T19:45:42.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.231+0000 7fe9fa59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe9fc0a98f0 0x7fe9fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58206/0 (socket says 172.21.15.107:58206) 2024-03-25T19:45:42.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.231+0000 7fe9fa59c640 1 -- 172.21.15.107:0/3098734958 learned_addr learned my addr 172.21.15.107:0/3098734958 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:42.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.241+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312730666 0 0) 0x7fe9fc142520 con 0x7fe9fc0a98f0 2024-03-25T19:45:42.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.241+0000 7fe9f959a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:42.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.241+0000 7fe9f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:42.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.241+0000 7fe9f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:42.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.241+0000 7fe9f959a640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:45:42.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.241+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e4003520 con 0x7fe9fc0a98f0 2024-03-25T19:45:42.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.249+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401482012 0 0) 0x7fe9fc0b28a0 con 0x7fe9fc13f710 2024-03-25T19:45:42.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.249+0000 7fe9f959a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:42.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.249+0000 7fe9f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:42.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.249+0000 7fe9f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:42.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.249+0000 7fe9f959a640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:45:42.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.249+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc142520 con 0x7fe9fc13f710 2024-03-25T19:45:42.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.257+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863007943 0 0) 0x7fe9fc0b2a30 con 0x7fe9fc1430c0 2024-03-25T19:45:42.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.257+0000 7fe9f959a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:42.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.257+0000 7fe9f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:42.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.257+0000 7fe9f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:42.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.257+0000 7fe9f959a640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:45:42.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.257+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc0b28a0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1506944165 0 0) 0x7fe9fc0b28a0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 >> v1:172.21.15.195:6789/0 conn(0x7fe9fc13f710 legacy=0x7fe9fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:42.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 >> v1:172.21.15.107:6789/0 conn(0x7fe9fc0a98f0 legacy=0x7fe9fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:42.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 10 monclient: _finish_hunting 0 2024-03-25T19:45:42.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 1 monclient: found mon.noname-b 2024-03-25T19:45:42.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:42.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc0b44c0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 10 monclient: _finish_auth 0 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 10 monclient: _check_auth_tickets 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe9f4001930 con 0x7fe9fc1430c0 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fea009e8640 5 monclient: authenticate success, global_id 4667 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fea009e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fe9f959a640 10 monclient: dump: 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:42.268 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:42.269 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:42.269 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:42.269 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:42.269 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:42.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.267+0000 7fea009e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:42.322 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9f4001a90 con 0x7fe9fc1430c0 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fe9f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fe9f959a640 1 -- 172.21.15.107:0/3098734958 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe9f4001ee0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fe9f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fe9f959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fe9f959a640 10 monclient: dump: 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:42.323 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:42.324 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:42.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fea009e8640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:42.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fea009e8640 4 set_mon_vals no callback set 2024-03-25T19:45:42.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fea009e8640 10 monclient: shutdown 2024-03-25T19:45:42.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fea009e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:42.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fea009e8640 1 -- 172.21.15.107:0/3098734958 >> v1:172.21.15.107:6790/0 conn(0x7fe9fc1430c0 legacy=0x7fe9fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:42.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fea009e8640 1 -- 172.21.15.107:0/3098734958 shutdown_connections 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fea009e8640 1 -- 172.21.15.107:0/3098734958 >> 172.21.15.107:0/3098734958 conn(0x7fe9fc0b7ea0 msgr2=0x7fe9fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fea009e8640 1 -- 172.21.15.107:0/3098734958 shutdown_connections 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.323+0000 7fea009e8640 1 -- 172.21.15.107:0/3098734958 wait complete. 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.324+0000 7fea009e8640 10 monclient: build_initial_monmap 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.324+0000 7fea009e8640 10 monclient: monmap: 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:42.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 1 librados: starting msgr at 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 1 librados: starting objecter 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 1 Processor -- start 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 1 -- start start 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 1 librados: setting wanted keys 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 1 librados: calling monclient init 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 10 monclient: init 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 10 monclient(hunting): picked mon.noname-b con 0x7fe9fc0a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 10 monclient(hunting): picked mon.noname-a con 0x7fe9fc13f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 10 monclient(hunting): picked mon.noname-c con 0x7fe9fc1430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.325+0000 7fea009e8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc1395e0 con 0x7fe9fc13f710 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.326+0000 7fea009e8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc1398c0 con 0x7fe9fc0a98f0 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.326+0000 7fea009e8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc0b0cc0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.326+0000 7fea009e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.326+0000 7fea009e8640 10 monclient(hunting): authenticate will time out at 5134.666782s 2024-03-25T19:45:42.328 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.326+0000 7fe9f9d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe9fc13f710 0x7fe9fc0affb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58260/0 (socket says 172.21.15.107:58260) 2024-03-25T19:45:42.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.326+0000 7fe9f9d9b640 1 -- 172.21.15.107:0/149865988 learned_addr learned my addr 172.21.15.107:0/149865988 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:42.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.326+0000 7fe9eaffd640 10 client.?.objecter ms_handle_connect 0x7fe9fc13f710 2024-03-25T19:45:42.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.326+0000 7fe9eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:42.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.326+0000 7fe9eaffd640 10 client.?.objecter ms_handle_connect 0x7fe9fc0a98f0 2024-03-25T19:45:42.329 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.326+0000 7fe9eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:42.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.349+0000 7fe9eaffd640 10 client.?.objecter ms_handle_connect 0x7fe9fc1430c0 2024-03-25T19:45:42.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.349+0000 7fe9eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:42.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.363+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321407039 0 0) 0x7fe9fc0b0cc0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.363+0000 7fe9eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:42.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.363+0000 7fe9eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:42.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.363+0000 7fe9eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:42.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.363+0000 7fe9eaffd640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:45:42.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.363+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9dc004eb0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.378+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2852042991 0 0) 0x7fe9dc004eb0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.378+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9fc0b0cc0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.378+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe9f4001de0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.378+0000 7fe9eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.378+0000 7fe9eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.378+0000 7fe9eaffd640 10 monclient(hunting): dump: 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:42.378 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:42.379 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:42.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.378+0000 7fe9eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:42.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.387+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777230984 0 0) 0x7fe9fc1398c0 con 0x7fe9fc0a98f0 2024-03-25T19:45:42.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.387+0000 7fe9eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:42.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.387+0000 7fe9eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:42.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.387+0000 7fe9eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:42.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.387+0000 7fe9eaffd640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:45:42.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.387+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9dc004eb0 con 0x7fe9fc0a98f0 2024-03-25T19:45:42.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3228690893 0 0) 0x7fe9fc0b0cc0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 >> v1:172.21.15.195:6789/0 conn(0x7fe9fc0a98f0 legacy=0x7fe9fc1d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 >> v1:172.21.15.107:6789/0 conn(0x7fe9fc13f710 legacy=0x7fe9fc0affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fe9eaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fe9eaffd640 1 monclient: found mon.c 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fe9eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fe9eaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc0b0e50 con 0x7fe9fc1430c0 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fe9eaffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fe9eaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fe9eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fea009e8640 5 monclient: authenticate success, global_id 4670 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fea009e8640 10 monclient: _renew_subs 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fea009e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:42.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fea009e8640 1 -- 172.21.15.107:0/149865988 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9fc0b06c0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fea009e8640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:42.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fea009e8640 10 monclient: _renew_subs 2024-03-25T19:45:42.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fea009e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:42.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fea009e8640 1 -- 172.21.15.107:0/149865988 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9fc1d7b00 con 0x7fe9fc1430c0 2024-03-25T19:45:42.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.403+0000 7fea009e8640 1 librados: init done 2024-03-25T19:45:42.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.405+0000 7fea009e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:42.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.405+0000 7fea009e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:42.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.405+0000 7fea009e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:42.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.405+0000 7fea009e8640 1 -- 172.21.15.107:0/149865988 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9fc0b1060 con 0x7fe9fc1430c0 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.723+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9f4002060 con 0x7fe9fc1430c0 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.723+0000 7fe9eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.723+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fe9f4003dc0 con 0x7fe9fc1430c0 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.723+0000 7fe9eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.723+0000 7fe9eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.723+0000 7fe9eaffd640 10 monclient: dump: 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:42.723 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:42.724 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:42.724 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:42.724 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:42.724 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:42.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.723+0000 7fe9f8d99640 4 set_mon_vals no callback set 2024-03-25T19:45:42.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.724+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fe9f4019d90 con 0x7fe9fc1430c0 2024-03-25T19:45:42.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.725+0000 7fe9fa59c640 10 monclient: get_auth_request con 0x7fe9dc06dd40 auth_method 0 2024-03-25T19:45:42.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.725+0000 7fe9eaffd640 10 client.4670.objecter ms_handle_connect 0x7fe9dc06dd40 2024-03-25T19:45:42.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.731+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7fe9f4053910 con 0x7fe9fc1430c0 2024-03-25T19:45:42.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.731+0000 7fe9eaffd640 10 client.4670.objecter ms_dispatch 0x7fe9fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:42.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.731+0000 7fe9eaffd640 3 client.4670.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:42.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.731+0000 7fe9eaffd640 3 client.4670.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:42.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.731+0000 7fe9eaffd640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-25T19:45:42.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.731+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9f404f250 con 0x7fe9fc1430c0 2024-03-25T19:45:42.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.731+0000 7fe9eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:42.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.731+0000 7fe9eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:42.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.976+0000 7fea009e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:42.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.976+0000 7fea009e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:42.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.976+0000 7fea009e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:45:42.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:42.976+0000 7fea009e8640 1 -- 172.21.15.107:0/149865988 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9fc0b3ef0 con 0x7fe9fc1430c0 2024-03-25T19:45:43.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.030+0000 7fe9eaffd640 1 -- 172.21.15.107:0/149865988 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7fe9f4051350 con 0x7fe9fc1430c0 2024-03-25T19:45:43.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.030+0000 7fe9eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:43.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.030+0000 7fe9eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:43.030 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:45:43.030 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.032+0000 7fea009e8640 10 librados: watch_flush enter 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.032+0000 7fea009e8640 10 librados: watch_flush exit 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.032+0000 7fea009e8640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.032+0000 7fea009e8640 10 client.4670.objecter successfully canceled tick 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.032+0000 7fea009e8640 1 -- 172.21.15.107:0/149865988 >> v1:172.21.15.195:6800/2984408421 conn(0x7fe9dc06dd40 legacy=0x7fe9dc070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.032+0000 7fea009e8640 10 monclient: shutdown 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.032+0000 7fea009e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.032+0000 7fea009e8640 1 -- 172.21.15.107:0/149865988 >> v1:172.21.15.107:6790/0 conn(0x7fe9fc1430c0 legacy=0x7fe9fc1d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.032+0000 7fea009e8640 1 -- 172.21.15.107:0/149865988 shutdown_connections 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.032+0000 7fea009e8640 1 -- 172.21.15.107:0/149865988 >> 172.21.15.107:0/149865988 conn(0x7fe9fc0b7ea0 msgr2=0x7fe9fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.032+0000 7fea009e8640 1 -- 172.21.15.107:0/149865988 shutdown_connections 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.033+0000 7fea009e8640 1 -- 172.21.15.107:0/149865988 wait complete. 2024-03-25T19:45:43.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:43.033+0000 7fea009e8640 1 librados: shutdown 2024-03-25T19:45:43.044 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:49.850 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:49.968 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.968+0000 7f86374d6640 10 monclient: get_monmap_and_config 2024-03-25T19:45:49.968 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.968+0000 7f86374d6640 10 monclient: build_initial_monmap 2024-03-25T19:45:49.968 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.968+0000 7f86374d6640 10 monclient: monmap: 2024-03-25T19:45:49.968 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:49.968 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:49.968 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:45:49.969252+0000 2024-03-25T19:45:49.968 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:45:49.969252+0000 2024-03-25T19:45:49.968 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:49.969 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:49.969 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:49.969 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:45:49.969 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:45:49.969 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:49.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.968+0000 7f86374d6640 1 Processor -- start 2024-03-25T19:45:49.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 1 -- start start 2024-03-25T19:45:49.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 10 monclient: init 2024-03-25T19:45:49.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:49.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:49.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 10 monclient(hunting): picked mon.noname-c con 0x7f86300b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:49.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 10 monclient(hunting): picked mon.noname-b con 0x7f86300b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:49.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 10 monclient(hunting): picked mon.noname-a con 0x7f863014be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:49.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86300af940 con 0x7f863014be90 2024-03-25T19:45:49.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86300b0a70 con 0x7f86300b4d60 2024-03-25T19:45:49.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86300b0d50 con 0x7f86300b1730 2024-03-25T19:45:49.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:49.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.969+0000 7f86374d6640 10 monclient(hunting): authenticate will time out at 5142.310500s 2024-03-25T19:45:49.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.978+0000 7f8635a73640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f863014be90 0x7f863014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34526/0 (socket says 172.21.15.107:34526) 2024-03-25T19:45:49.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.978+0000 7f8635a73640 1 -- 172.21.15.107:0/3491525164 learned_addr learned my addr 172.21.15.107:0/3491525164 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:49.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.984+0000 7f8627fff640 1 -- 172.21.15.107:0/3491525164 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421234556 0 0) 0x7f86300af940 con 0x7f863014be90 2024-03-25T19:45:49.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.984+0000 7f8627fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:49.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.984+0000 7f8627fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:49.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.984+0000 7f8627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:49.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.984+0000 7f8627fff640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:45:49.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.984+0000 7f8627fff640 1 -- 172.21.15.107:0/3491525164 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861c003520 con 0x7f863014be90 2024-03-25T19:45:49.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.996+0000 7f8627fff640 1 -- 172.21.15.107:0/3491525164 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501301336 0 0) 0x7f86300b0a70 con 0x7f86300b4d60 2024-03-25T19:45:49.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.996+0000 7f8627fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:49.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.996+0000 7f8627fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:49.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.996+0000 7f8627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:49.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.996+0000 7f8627fff640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:45:49.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.996+0000 7f8627fff640 1 -- 172.21.15.107:0/3491525164 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86300af940 con 0x7f86300b4d60 2024-03-25T19:45:49.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 1 -- 172.21.15.107:0/3491525164 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3174243889 0 0) 0x7f861c003520 con 0x7f863014be90 2024-03-25T19:45:49.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 1 -- 172.21.15.107:0/3491525164 >> v1:172.21.15.107:6790/0 conn(0x7f86300b1730 legacy=0x7f86300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:49.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 1 -- 172.21.15.107:0/3491525164 >> v1:172.21.15.195:6789/0 conn(0x7f86300b4d60 legacy=0x7f863013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:49.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 1 monclient: found mon.noname-a 2024-03-25T19:45:49.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:49.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:45:49.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 1 -- 172.21.15.107:0/3491525164 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86300b1030 con 0x7f863014be90 2024-03-25T19:45:49.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 10 monclient: _finish_auth 0 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 1 -- 172.21.15.107:0/3491525164 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8628001a10 con 0x7f863014be90 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f8627fff640 10 monclient: dump: 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:49.998 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:49.999 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:49.999 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f86374d6640 5 monclient: authenticate success, global_id 4668 2024-03-25T19:45:49.999 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:49.997+0000 7f86374d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:50.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.007+0000 7f8627fff640 1 -- 172.21.15.107:0/3491525164 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8628001b70 con 0x7f863014be90 2024-03-25T19:45:50.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.007+0000 7f8627fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:50.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.007+0000 7f8627fff640 1 -- 172.21.15.107:0/3491525164 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8628001f80 con 0x7f863014be90 2024-03-25T19:45:50.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.007+0000 7f86374d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:50.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.007+0000 7f86374d6640 4 set_mon_vals no callback set 2024-03-25T19:45:50.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.007+0000 7f8627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:50.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.007+0000 7f8627fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:50.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.007+0000 7f8627fff640 10 monclient: dump: 2024-03-25T19:45:50.008 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:50.008 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:50.008 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:50.008 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:50.008 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:50.008 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:50.008 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:50.008 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:50.008 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:50.008 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:50.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.008+0000 7f86374d6640 10 monclient: shutdown 2024-03-25T19:45:50.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.008+0000 7f86374d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:50.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.008+0000 7f86374d6640 1 -- 172.21.15.107:0/3491525164 >> v1:172.21.15.107:6789/0 conn(0x7f863014be90 legacy=0x7f863014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:50.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.008+0000 7f86374d6640 1 -- 172.21.15.107:0/3491525164 shutdown_connections 2024-03-25T19:45:50.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.008+0000 7f86374d6640 1 -- 172.21.15.107:0/3491525164 >> 172.21.15.107:0/3491525164 conn(0x7f86300617a0 msgr2=0x7f8630061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:50.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.008+0000 7f86374d6640 1 -- 172.21.15.107:0/3491525164 shutdown_connections 2024-03-25T19:45:50.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.008+0000 7f86374d6640 1 -- 172.21.15.107:0/3491525164 wait complete. 2024-03-25T19:45:50.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.009+0000 7f86374d6640 10 monclient: build_initial_monmap 2024-03-25T19:45:50.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.009+0000 7f86374d6640 10 monclient: monmap: 2024-03-25T19:45:50.010 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:50.011 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:50.011 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:45:50.011 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:45:50.011 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:50.011 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:45:50.011 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:50.012 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:45:50.012 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:45:50.012 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:50.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.009+0000 7f86374d6640 1 librados: starting msgr at 2024-03-25T19:45:50.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.009+0000 7f86374d6640 1 librados: starting objecter 2024-03-25T19:45:50.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.010+0000 7f86374d6640 1 Processor -- start 2024-03-25T19:45:50.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.010+0000 7f86374d6640 1 -- start start 2024-03-25T19:45:50.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.010+0000 7f86374d6640 1 librados: setting wanted keys 2024-03-25T19:45:50.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.010+0000 7f86374d6640 1 librados: calling monclient init 2024-03-25T19:45:50.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.010+0000 7f86374d6640 10 monclient: init 2024-03-25T19:45:50.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.010+0000 7f86374d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:50.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.010+0000 7f86374d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:50.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.010+0000 7f86374d6640 10 monclient(hunting): picked mon.noname-c con 0x7f86300b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:50.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.010+0000 7f86374d6640 10 monclient(hunting): picked mon.noname-b con 0x7f86300b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:50.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.010+0000 7f86374d6640 10 monclient(hunting): picked mon.noname-a con 0x7f863014be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:50.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.010+0000 7f86374d6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8630135c30 con 0x7f863014be90 2024-03-25T19:45:50.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.011+0000 7f86374d6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86301db2a0 con 0x7f86300b4d60 2024-03-25T19:45:50.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.011+0000 7f86374d6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8630136b60 con 0x7f86300b1730 2024-03-25T19:45:50.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.011+0000 7f86374d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:50.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.011+0000 7f86374d6640 10 monclient(hunting): authenticate will time out at 5142.351776s 2024-03-25T19:45:50.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.018+0000 7f8635a73640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f863014be90 0x7f86301e2ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34564/0 (socket says 172.21.15.107:34564) 2024-03-25T19:45:50.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.018+0000 7f8635a73640 1 -- 172.21.15.107:0/3020652806 learned_addr learned my addr 172.21.15.107:0/3020652806 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:50.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.018+0000 7f8625ffb640 10 client.?.objecter ms_handle_connect 0x7f863014be90 2024-03-25T19:45:50.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.018+0000 7f8625ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:50.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.027+0000 7f8625ffb640 10 client.?.objecter ms_handle_connect 0x7f86300b1730 2024-03-25T19:45:50.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.027+0000 7f8625ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:50.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.029+0000 7f8625ffb640 10 client.?.objecter ms_handle_connect 0x7f86300b4d60 2024-03-25T19:45:50.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.029+0000 7f8625ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:50.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.031+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192475245 0 0) 0x7f8630135c30 con 0x7f863014be90 2024-03-25T19:45:50.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.031+0000 7f8625ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:50.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.031+0000 7f8625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:50.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.031+0000 7f8625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:50.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.031+0000 7f8625ffb640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:45:50.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.031+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8614004eb0 con 0x7f863014be90 2024-03-25T19:45:50.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.041+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093494013 0 0) 0x7f8630136b60 con 0x7f86300b1730 2024-03-25T19:45:50.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.041+0000 7f8625ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:50.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.041+0000 7f8625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:50.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.041+0000 7f8625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:50.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.041+0000 7f8625ffb640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:45:50.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.041+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8630135c30 con 0x7f86300b1730 2024-03-25T19:45:50.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.046+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368621042 0 0) 0x7f86301db2a0 con 0x7f86300b4d60 2024-03-25T19:45:50.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.046+0000 7f8625ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:50.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.046+0000 7f8625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:50.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.046+0000 7f8625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:50.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.046+0000 7f8625ffb640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:45:50.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.046+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8630136b60 con 0x7f86300b4d60 2024-03-25T19:45:50.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.047+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1536974293 0 0) 0x7f8614004eb0 con 0x7f863014be90 2024-03-25T19:45:50.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.047+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86301db2a0 con 0x7f863014be90 2024-03-25T19:45:50.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.047+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8628001ec0 con 0x7f863014be90 2024-03-25T19:45:50.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.047+0000 7f8625ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:50.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.047+0000 7f8625ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:50.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.047+0000 7f8625ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:50.048 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:50.048 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:50.048 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:50.048 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:50.048 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:50.048 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:50.048 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:50.048 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:50.048 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:50.048 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:50.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.047+0000 7f8625ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:50.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 936497715 0 0) 0x7f86301db2a0 con 0x7f863014be90 2024-03-25T19:45:50.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 >> v1:172.21.15.107:6790/0 conn(0x7f86300b1730 legacy=0x7f86301dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:50.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 >> v1:172.21.15.195:6789/0 conn(0x7f86300b4d60 legacy=0x7f863012fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:50.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f8625ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f8625ffb640 1 monclient: found mon.a 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f8625ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f8625ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8630136cf0 con 0x7f863014be90 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f8625ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f8625ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f8625ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f86374d6640 5 monclient: authenticate success, global_id 4671 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f86374d6640 10 monclient: _renew_subs 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f86374d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f86374d6640 1 -- 172.21.15.107:0/3020652806 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86301360c0 con 0x7f863014be90 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f86374d6640 10 client.4671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f86374d6640 10 monclient: _renew_subs 2024-03-25T19:45:50.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f86374d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:45:50.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.055+0000 7f86374d6640 1 -- 172.21.15.107:0/3020652806 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86301457d0 con 0x7f863014be90 2024-03-25T19:45:50.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.056+0000 7f86374d6640 1 librados: init done 2024-03-25T19:45:50.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.057+0000 7f86374d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:50.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.057+0000 7f86374d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:50.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.057+0000 7f86374d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:45:50.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.057+0000 7f86374d6640 1 -- 172.21.15.107:0/3020652806 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8630135de0 con 0x7f863014be90 2024-03-25T19:45:50.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.351+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8628004e10 con 0x7f863014be90 2024-03-25T19:45:50.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.351+0000 7f8625ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:50.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.351+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f86280015d0 con 0x7f863014be90 2024-03-25T19:45:50.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.351+0000 7f8625ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:50.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.351+0000 7f8625ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:50.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.351+0000 7f8625ffb640 10 monclient: dump: 2024-03-25T19:45:50.351 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:50.352 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:50.352 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:50.352 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:50.352 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:50.352 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:50.352 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:50.352 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:50.352 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:50.352 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:50.352 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.351+0000 7f86277fe640 4 set_mon_vals no callback set 2024-03-25T19:45:50.352 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.351+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f862801adf0 con 0x7f863014be90 2024-03-25T19:45:50.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.352+0000 7f8635272640 10 monclient: get_auth_request con 0x7f861406dda0 auth_method 0 2024-03-25T19:45:50.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.352+0000 7f8625ffb640 10 client.4671.objecter ms_handle_connect 0x7f861406dda0 2024-03-25T19:45:50.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.357+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 305850033 0 0) 0x7f86280547f0 con 0x7f863014be90 2024-03-25T19:45:50.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.357+0000 7f8625ffb640 10 client.4671.objecter ms_dispatch 0x7f8630000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:50.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.357+0000 7f8625ffb640 3 client.4671.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:50.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.357+0000 7f8625ffb640 3 client.4671.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:50.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.357+0000 7f8625ffb640 20 client.4671.objecter dump_active .. 0 homeless 2024-03-25T19:45:50.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.357+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8628050130 con 0x7f863014be90 2024-03-25T19:45:50.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.357+0000 7f8625ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:50.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.357+0000 7f8625ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:50.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.587+0000 7f86374d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:50.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.587+0000 7f86374d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:50.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.587+0000 7f86374d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:45:50.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.587+0000 7f86374d6640 1 -- 172.21.15.107:0/3020652806 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8630136cf0 con 0x7f863014be90 2024-03-25T19:45:50.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.712+0000 7f8625ffb640 1 -- 172.21.15.107:0/3020652806 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2339568975) 0x7f8630136cf0 con 0x7f863014be90 2024-03-25T19:45:50.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.712+0000 7f8625ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:50.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.712+0000 7f8625ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:50.713 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:45:50.713 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:35:51.640883+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:51.640883+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:50.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 10 librados: watch_flush enter 2024-03-25T19:45:50.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 10 librados: watch_flush exit 2024-03-25T19:45:50.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 20 client.4671.objecter shutdown clearing up homeless session... 2024-03-25T19:45:50.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 10 client.4671.objecter successfully canceled tick 2024-03-25T19:45:50.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 1 -- 172.21.15.107:0/3020652806 >> v1:172.21.15.195:6800/2984408421 conn(0x7f861406dda0 legacy=0x7f8614070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:50.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 10 monclient: shutdown 2024-03-25T19:45:50.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:50.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 1 -- 172.21.15.107:0/3020652806 >> v1:172.21.15.107:6789/0 conn(0x7f863014be90 legacy=0x7f86301e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:50.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 1 -- 172.21.15.107:0/3020652806 shutdown_connections 2024-03-25T19:45:50.716 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 1 -- 172.21.15.107:0/3020652806 >> 172.21.15.107:0/3020652806 conn(0x7f86300617a0 msgr2=0x7f8630061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:50.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 1 -- 172.21.15.107:0/3020652806 shutdown_connections 2024-03-25T19:45:50.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 1 -- 172.21.15.107:0/3020652806 wait complete. 2024-03-25T19:45:50.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:50.715+0000 7f86374d6640 1 librados: shutdown 2024-03-25T19:45:50.727 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:51.613 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:10:21.929 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:51.617 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:10:21.934 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:51.642 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:10:21.958 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:51.665 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:10:21.952 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:51.671 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:10:21.958 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:51.681 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:10:21.998 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:51.690 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:10:21.977 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:51.693 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:10:22.009 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:51.716 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:10:22.032 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:57.534 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:57.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.648+0000 7f5dd1bb6640 10 monclient: get_monmap_and_config 2024-03-25T19:45:57.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.648+0000 7f5dd1bb6640 10 monclient: build_initial_monmap 2024-03-25T19:45:57.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.648+0000 7f5dd1bb6640 10 monclient: monmap: 2024-03-25T19:45:57.649 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:57.649 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:57.649 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:45:57.649623+0000 2024-03-25T19:45:57.649 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:45:57.649623+0000 2024-03-25T19:45:57.650 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:57.650 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:57.650 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:57.650 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:45:57.650 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:45:57.650 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:57.652 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.649+0000 7f5dd1bb6640 1 Processor -- start 2024-03-25T19:45:57.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.649+0000 7f5dd1bb6640 1 -- start start 2024-03-25T19:45:57.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.649+0000 7f5dd1bb6640 10 monclient: init 2024-03-25T19:45:57.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.650+0000 7f5dd1bb6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:57.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.650+0000 7f5dd1bb6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:57.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.650+0000 7f5dd1bb6640 10 monclient(hunting): picked mon.noname-c con 0x7f5dcc061830 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:57.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.650+0000 7f5dd1bb6640 10 monclient(hunting): picked mon.noname-a con 0x7f5dcc1400e0 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:57.653 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.650+0000 7f5dd1bb6640 10 monclient(hunting): picked mon.noname-b con 0x7f5dcc1499d0 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:57.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.650+0000 7f5dd1bb6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc072600 con 0x7f5dcc1400e0 2024-03-25T19:45:57.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.650+0000 7f5dd1bb6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc140c80 con 0x7f5dcc061830 2024-03-25T19:45:57.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.650+0000 7f5dd1bb6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc072830 con 0x7f5dcc1499d0 2024-03-25T19:45:57.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.650+0000 7f5dd1bb6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:57.654 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.650+0000 7f5dd1bb6640 10 monclient(hunting): authenticate will time out at 5149.991639s 2024-03-25T19:45:57.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.661+0000 7f5dcbfff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f5dcc1499d0 0x7f5dcc14bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38860/0 (socket says 172.21.15.107:38860) 2024-03-25T19:45:57.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.661+0000 7f5dcbfff640 1 -- 172.21.15.107:0/951303226 learned_addr learned my addr 172.21.15.107:0/951303226 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:57.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.668+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752170716 0 0) 0x7f5dcc072830 con 0x7f5dcc1499d0 2024-03-25T19:45:57.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.669+0000 7f5dca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:57.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.669+0000 7f5dca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:57.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.669+0000 7f5dca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:57.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.669+0000 7f5dca7fc640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:45:57.669 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.669+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db4003520 con 0x7f5dcc1499d0 2024-03-25T19:45:57.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.673+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222674806 0 0) 0x7f5dcc072600 con 0x7f5dcc1400e0 2024-03-25T19:45:57.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.673+0000 7f5dca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:57.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.673+0000 7f5dca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:57.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.673+0000 7f5dca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:57.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.673+0000 7f5dca7fc640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:45:57.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.673+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dcc072830 con 0x7f5dcc1400e0 2024-03-25T19:45:57.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.674+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769789772 0 0) 0x7f5dcc140c80 con 0x7f5dcc061830 2024-03-25T19:45:57.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.674+0000 7f5dca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:57.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.674+0000 7f5dca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:57.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.674+0000 7f5dca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:57.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.674+0000 7f5dca7fc640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:45:57.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.674+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dcc072600 con 0x7f5dcc061830 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677228850 0 0) 0x7f5dcc072600 con 0x7f5dcc061830 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 >> v1:172.21.15.107:6790/0 conn(0x7f5dcc1499d0 legacy=0x7f5dcc14bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 >> v1:172.21.15.107:6789/0 conn(0x7f5dcc1400e0 legacy=0x7f5dcc140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 1 monclient: found mon.noname-c 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc072b10 con 0x7f5dcc061830 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:57.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5dbc0015b0 con 0x7f5dcc061830 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dd1bb6640 5 monclient: authenticate success, global_id 4666 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dd1bb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dca7fc640 10 monclient: dump: 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:57.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.686+0000 7f5dd1bb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:57.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.694+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dbc001710 con 0x7f5dcc061830 2024-03-25T19:45:57.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.694+0000 7f5dca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:57.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.694+0000 7f5dca7fc640 1 -- 172.21.15.107:0/951303226 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5dbc001b60 con 0x7f5dcc061830 2024-03-25T19:45:57.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.694+0000 7f5dd1bb6640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:57.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.694+0000 7f5dd1bb6640 4 set_mon_vals no callback set 2024-03-25T19:45:57.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.694+0000 7f5dca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.694+0000 7f5dca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.694+0000 7f5dca7fc640 10 monclient: dump: 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:57.695 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.695+0000 7f5dd1bb6640 10 monclient: shutdown 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.695+0000 7f5dd1bb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.695+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/951303226 >> v1:172.21.15.195:6789/0 conn(0x7f5dcc061830 legacy=0x7f5dcc13d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.695+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/951303226 shutdown_connections 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.695+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/951303226 >> 172.21.15.107:0/951303226 conn(0x7f5dcc0b0ef0 msgr2=0x7f5dcc0b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.695+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/951303226 shutdown_connections 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.696+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/951303226 wait complete. 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.696+0000 7f5dd1bb6640 10 monclient: build_initial_monmap 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.696+0000 7f5dd1bb6640 10 monclient: monmap: 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:57.697 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.697+0000 7f5dd1bb6640 1 librados: starting msgr at 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.697+0000 7f5dd1bb6640 1 librados: starting objecter 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.697+0000 7f5dd1bb6640 1 Processor -- start 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.697+0000 7f5dd1bb6640 1 -- start start 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.697+0000 7f5dd1bb6640 1 librados: setting wanted keys 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.697+0000 7f5dd1bb6640 1 librados: calling monclient init 2024-03-25T19:45:57.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.697+0000 7f5dd1bb6640 10 monclient: init 2024-03-25T19:45:57.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.698+0000 7f5dd1bb6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:57.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.698+0000 7f5dd1bb6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:57.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.698+0000 7f5dd1bb6640 10 monclient(hunting): picked mon.noname-c con 0x7f5dcc061830 addr v1:172.21.15.107:6790/0 2024-03-25T19:45:57.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.698+0000 7f5dd1bb6640 10 monclient(hunting): picked mon.noname-b con 0x7f5dcc1400e0 addr v1:172.21.15.195:6789/0 2024-03-25T19:45:57.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.698+0000 7f5dd1bb6640 10 monclient(hunting): picked mon.noname-a con 0x7f5dcc1499d0 addr v1:172.21.15.107:6789/0 2024-03-25T19:45:57.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.698+0000 7f5dd1bb6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc0aabd0 con 0x7f5dcc1499d0 2024-03-25T19:45:57.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.698+0000 7f5dd1bb6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc139940 con 0x7f5dcc1400e0 2024-03-25T19:45:57.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.698+0000 7f5dd1bb6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc139ad0 con 0x7f5dcc061830 2024-03-25T19:45:57.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.698+0000 7f5dd1bb6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:57.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.698+0000 7f5dd1bb6640 10 monclient(hunting): authenticate will time out at 5150.038916s 2024-03-25T19:45:57.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.709+0000 7f5dcb7fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f5dcc061830 0x7f5dcc1d5440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38906/0 (socket says 172.21.15.107:38906) 2024-03-25T19:45:57.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.709+0000 7f5dcb7fe640 1 -- 172.21.15.107:0/2414925326 learned_addr learned my addr 172.21.15.107:0/2414925326 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:45:57.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.710+0000 7f5db3fff640 10 client.?.objecter ms_handle_connect 0x7f5dcc061830 2024-03-25T19:45:57.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.710+0000 7f5db3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:57.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.719+0000 7f5db3fff640 10 client.?.objecter ms_handle_connect 0x7f5dcc1400e0 2024-03-25T19:45:57.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.719+0000 7f5db3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:57.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.719+0000 7f5db3fff640 10 client.?.objecter ms_handle_connect 0x7f5dcc1499d0 2024-03-25T19:45:57.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.719+0000 7f5db3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:57.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.720+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781392262 0 0) 0x7f5dcc139ad0 con 0x7f5dcc061830 2024-03-25T19:45:57.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.720+0000 7f5db3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:57.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.720+0000 7f5db3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:57.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.720+0000 7f5db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:57.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.720+0000 7f5db3fff640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:45:57.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.720+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db8004eb0 con 0x7f5dcc061830 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.725+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276493888 0 0) 0x7f5dcc0aabd0 con 0x7f5dcc1499d0 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.725+0000 7f5db3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.725+0000 7f5db3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.725+0000 7f5db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.725+0000 7f5db3fff640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:45:57.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.725+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dcc139ad0 con 0x7f5dcc1499d0 2024-03-25T19:45:57.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.730+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483295409 0 0) 0x7f5dcc139940 con 0x7f5dcc1400e0 2024-03-25T19:45:57.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.730+0000 7f5db3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:57.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.730+0000 7f5db3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:57.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.730+0000 7f5db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:57.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.730+0000 7f5db3fff640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:45:57.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.730+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dcc0aabd0 con 0x7f5dcc1400e0 2024-03-25T19:45:57.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.735+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1196730376 0 0) 0x7f5dcc139ad0 con 0x7f5dcc1499d0 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.735+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dcc139940 con 0x7f5dcc1499d0 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.735+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5dc4001ab0 con 0x7f5dcc1499d0 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.735+0000 7f5db3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.735+0000 7f5db3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.735+0000 7f5db3fff640 10 monclient(hunting): dump: 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:57.735 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:57.736 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:57.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.735+0000 7f5db3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:57.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 983216558 0 0) 0x7f5dcc139940 con 0x7f5dcc1499d0 2024-03-25T19:45:57.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 >> v1:172.21.15.107:6790/0 conn(0x7f5dcc061830 legacy=0x7f5dcc1d5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 >> v1:172.21.15.195:6789/0 conn(0x7f5dcc1400e0 legacy=0x7f5dcc1d5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5db3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:57.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5db3fff640 1 monclient: found mon.a 2024-03-25T19:45:57.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5db3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:57.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5db3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:45:57.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc1377b0 con 0x7f5dcc1499d0 2024-03-25T19:45:57.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5db3fff640 10 monclient: _finish_auth 0 2024-03-25T19:45:57.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5db3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:57.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5db3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:57.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5dd1bb6640 5 monclient: authenticate success, global_id 4677 2024-03-25T19:45:57.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5dd1bb6640 10 monclient: _renew_subs 2024-03-25T19:45:57.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5dd1bb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:45:57.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/2414925326 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dcc0a97e0 con 0x7f5dcc1499d0 2024-03-25T19:45:57.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5dd1bb6640 10 client.4677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:57.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5dd1bb6640 10 monclient: _renew_subs 2024-03-25T19:45:57.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5dd1bb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:45:57.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/2414925326 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dcc1e13e0 con 0x7f5dcc1499d0 2024-03-25T19:45:57.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.743+0000 7f5dd1bb6640 1 librados: init done 2024-03-25T19:45:57.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.744+0000 7f5dd1bb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:57.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.744+0000 7f5dd1bb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:57.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.744+0000 7f5dd1bb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:45:57.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:57.744+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/2414925326 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dcc137a00 con 0x7f5dcc1499d0 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.061+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dc4001f80 con 0x7f5dcc1499d0 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.061+0000 7f5db3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.061+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f5dc4004d00 con 0x7f5dcc1499d0 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.061+0000 7f5db3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.061+0000 7f5db3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.061+0000 7f5db3fff640 10 monclient: dump: 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:58.061 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:45:58.062 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:45:58.062 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:45:58.062 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:45:58.062 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:45:58.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.061+0000 7f5dc9ffb640 4 set_mon_vals no callback set 2024-03-25T19:45:58.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.062+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f5dc401acd0 con 0x7f5dcc1499d0 2024-03-25T19:45:58.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.062+0000 7f5dcb7fe640 10 monclient: get_auth_request con 0x7f5db806de20 auth_method 0 2024-03-25T19:45:58.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.062+0000 7f5db3fff640 10 client.4677.objecter ms_handle_connect 0x7f5db806de20 2024-03-25T19:45:58.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.067+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3155620770 0 0) 0x7f5dc4054920 con 0x7f5dcc1499d0 2024-03-25T19:45:58.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.067+0000 7f5db3fff640 10 client.4677.objecter ms_dispatch 0x7f5dcc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:45:58.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.067+0000 7f5db3fff640 3 client.4677.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:45:58.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.067+0000 7f5db3fff640 3 client.4677.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:45:58.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.067+0000 7f5db3fff640 20 client.4677.objecter dump_active .. 0 homeless 2024-03-25T19:45:58.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.067+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5dc4050260 con 0x7f5dcc1499d0 2024-03-25T19:45:58.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.067+0000 7f5db3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:58.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.067+0000 7f5db3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:58.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.311+0000 7f5dd1bb6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:58.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.311+0000 7f5dd1bb6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:58.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.311+0000 7f5dd1bb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:45:58.311 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.311+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/2414925326 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dcc0627b0 con 0x7f5dcc1499d0 2024-03-25T19:45:58.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.435+0000 7f5db3fff640 1 -- 172.21.15.107:0/2414925326 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2864573051) 0x7f5dc4052360 con 0x7f5dcc1499d0 2024-03-25T19:45:58.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.435+0000 7f5db3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:58.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.435+0000 7f5db3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:58.436 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:45:58.436 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:45:51.343698+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:51.343698+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:58.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 10 librados: watch_flush enter 2024-03-25T19:45:58.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 10 librados: watch_flush exit 2024-03-25T19:45:58.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 20 client.4677.objecter shutdown clearing up homeless session... 2024-03-25T19:45:58.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 10 client.4677.objecter successfully canceled tick 2024-03-25T19:45:58.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/2414925326 >> v1:172.21.15.195:6800/2984408421 conn(0x7f5db806de20 legacy=0x7f5db80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:58.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 10 monclient: shutdown 2024-03-25T19:45:58.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:58.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/2414925326 >> v1:172.21.15.107:6789/0 conn(0x7f5dcc1499d0 legacy=0x7f5dcc1dfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:58.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/2414925326 shutdown_connections 2024-03-25T19:45:58.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/2414925326 >> 172.21.15.107:0/2414925326 conn(0x7f5dcc0b0ef0 msgr2=0x7f5dcc13d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:58.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/2414925326 shutdown_connections 2024-03-25T19:45:58.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 1 -- 172.21.15.107:0/2414925326 wait complete. 2024-03-25T19:45:58.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:45:58.438+0000 7f5dd1bb6640 1 librados: shutdown 2024-03-25T19:45:58.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:05.257 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:05.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.374+0000 7f69e4055640 10 monclient: get_monmap_and_config 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.374+0000 7f69e4055640 10 monclient: build_initial_monmap 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.374+0000 7f69e4055640 10 monclient: monmap: 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:46:05.375027+0000 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:46:05.375027+0000 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:46:05.374 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:05.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.374+0000 7f69e4055640 1 Processor -- start 2024-03-25T19:46:05.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.374+0000 7f69e4055640 1 -- start start 2024-03-25T19:46:05.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.374+0000 7f69e4055640 10 monclient: init 2024-03-25T19:46:05.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.375+0000 7f69e4055640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:05.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.375+0000 7f69e4055640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:05.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.375+0000 7f69e4055640 10 monclient(hunting): picked mon.noname-a con 0x7f69dc0b10d0 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:05.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.375+0000 7f69e4055640 10 monclient(hunting): picked mon.noname-b con 0x7f69dc0b4360 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:05.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.375+0000 7f69e4055640 10 monclient(hunting): picked mon.noname-c con 0x7f69dc0a9470 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:05.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.375+0000 7f69e4055640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0a9ab0 con 0x7f69dc0b10d0 2024-03-25T19:46:05.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.375+0000 7f69e4055640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0b2400 con 0x7f69dc0a9470 2024-03-25T19:46:05.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.375+0000 7f69e4055640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0aa2a0 con 0x7f69dc0b4360 2024-03-25T19:46:05.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.375+0000 7f69e4055640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:05.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.375+0000 7f69e4055640 10 monclient(hunting): authenticate will time out at 5157.716546s 2024-03-25T19:46:05.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.384+0000 7f69e1df1640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f69dc0b10d0 0x7f69dc0b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39456/0 (socket says 172.21.15.107:39456) 2024-03-25T19:46:05.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.384+0000 7f69e1df1640 1 -- 172.21.15.107:0/1593382010 learned_addr learned my addr 172.21.15.107:0/1593382010 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:05.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.397+0000 7f69e0def640 1 -- 172.21.15.107:0/1593382010 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428966285 0 0) 0x7f69dc0a9ab0 con 0x7f69dc0b10d0 2024-03-25T19:46:05.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.397+0000 7f69e0def640 10 monclient(hunting): _negotiate 2024-03-25T19:46:05.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.397+0000 7f69e0def640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:05.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.397+0000 7f69e0def640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:05.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.397+0000 7f69e0def640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:46:05.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.397+0000 7f69e0def640 1 -- 172.21.15.107:0/1593382010 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c8003520 con 0x7f69dc0b10d0 2024-03-25T19:46:05.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.408+0000 7f69e0def640 1 -- 172.21.15.107:0/1593382010 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645761326 0 0) 0x7f69dc0aa2a0 con 0x7f69dc0b4360 2024-03-25T19:46:05.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.408+0000 7f69e0def640 10 monclient(hunting): _negotiate 2024-03-25T19:46:05.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.408+0000 7f69e0def640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:05.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.408+0000 7f69e0def640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:05.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.408+0000 7f69e0def640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:46:05.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.408+0000 7f69e0def640 1 -- 172.21.15.107:0/1593382010 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69dc0a9ab0 con 0x7f69dc0b4360 2024-03-25T19:46:05.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 1 -- 172.21.15.107:0/1593382010 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1202764316 0 0) 0x7f69dc0a9ab0 con 0x7f69dc0b4360 2024-03-25T19:46:05.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 1 -- 172.21.15.107:0/1593382010 >> v1:172.21.15.195:6789/0 conn(0x7f69dc0a9470 legacy=0x7f69dc14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 1 -- 172.21.15.107:0/1593382010 >> v1:172.21.15.107:6789/0 conn(0x7f69dc0b10d0 legacy=0x7f69dc0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 10 monclient: _finish_hunting 0 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 1 monclient: found mon.noname-b 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:46:05.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 1 -- 172.21.15.107:0/1593382010 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc0b14b0 con 0x7f69dc0b4360 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 10 monclient: _finish_auth 0 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 10 monclient: _check_auth_tickets 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 1 -- 172.21.15.107:0/1593382010 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f69cc001670 con 0x7f69dc0b4360 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e4055640 5 monclient: authenticate success, global_id 4682 2024-03-25T19:46:05.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e4055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:05.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:05.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:05.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e0def640 10 monclient: dump: 2024-03-25T19:46:05.422 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:05.422 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:05.423 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:05.423 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:05.423 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:05.423 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:05.423 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:05.423 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:05.424 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:05.424 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:05.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.419+0000 7f69e4055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:05.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.456+0000 7f69e0def640 1 -- 172.21.15.107:0/1593382010 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69cc0017d0 con 0x7f69dc0b4360 2024-03-25T19:46:05.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.456+0000 7f69e0def640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:05.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.456+0000 7f69e0def640 1 -- 172.21.15.107:0/1593382010 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f69cc001c20 con 0x7f69dc0b4360 2024-03-25T19:46:05.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.456+0000 7f69e0def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:05.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.456+0000 7f69e0def640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:05.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.456+0000 7f69e0def640 10 monclient: dump: 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.456+0000 7f69e4055640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:05.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.456+0000 7f69e4055640 4 set_mon_vals no callback set 2024-03-25T19:46:05.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.457+0000 7f69e4055640 10 monclient: shutdown 2024-03-25T19:46:05.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.457+0000 7f69e4055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:05.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.457+0000 7f69e4055640 1 -- 172.21.15.107:0/1593382010 >> v1:172.21.15.107:6790/0 conn(0x7f69dc0b4360 legacy=0x7f69dc0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.457+0000 7f69e4055640 1 -- 172.21.15.107:0/1593382010 shutdown_connections 2024-03-25T19:46:05.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.457+0000 7f69e4055640 1 -- 172.21.15.107:0/1593382010 >> 172.21.15.107:0/1593382010 conn(0x7f69dc0617a0 msgr2=0x7f69dc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:05.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.457+0000 7f69e4055640 1 -- 172.21.15.107:0/1593382010 shutdown_connections 2024-03-25T19:46:05.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.457+0000 7f69e4055640 1 -- 172.21.15.107:0/1593382010 wait complete. 2024-03-25T19:46:05.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.458+0000 7f69e4055640 10 monclient: build_initial_monmap 2024-03-25T19:46:05.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.458+0000 7f69e4055640 10 monclient: monmap: 2024-03-25T19:46:05.459 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.458+0000 7f69e4055640 1 librados: starting msgr at 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.458+0000 7f69e4055640 1 librados: starting objecter 2024-03-25T19:46:05.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.459+0000 7f69e4055640 1 Processor -- start 2024-03-25T19:46:05.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.459+0000 7f69e4055640 1 -- start start 2024-03-25T19:46:05.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.459+0000 7f69e4055640 1 librados: setting wanted keys 2024-03-25T19:46:05.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.459+0000 7f69e4055640 1 librados: calling monclient init 2024-03-25T19:46:05.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.459+0000 7f69e4055640 10 monclient: init 2024-03-25T19:46:05.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.459+0000 7f69e4055640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:05.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.459+0000 7f69e4055640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:05.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.460+0000 7f69e4055640 10 monclient(hunting): picked mon.noname-a con 0x7f69dc0a9470 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:05.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.460+0000 7f69e4055640 10 monclient(hunting): picked mon.noname-c con 0x7f69dc0b10d0 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:05.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.460+0000 7f69e4055640 10 monclient(hunting): picked mon.noname-b con 0x7f69dc0b4360 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:05.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.460+0000 7f69e4055640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc148370 con 0x7f69dc0a9470 2024-03-25T19:46:05.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.460+0000 7f69e4055640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0b0920 con 0x7f69dc0b4360 2024-03-25T19:46:05.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.460+0000 7f69e4055640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc1485c0 con 0x7f69dc0b10d0 2024-03-25T19:46:05.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.460+0000 7f69e4055640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:05.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.460+0000 7f69e4055640 10 monclient(hunting): authenticate will time out at 5157.800970s 2024-03-25T19:46:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.466+0000 7f69e25f2640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f69dc0b4360 0x7f69dc1e2b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:36572/0 (socket says 172.21.15.107:36572) 2024-03-25T19:46:05.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.466+0000 7f69e25f2640 1 -- 172.21.15.107:0/3724082632 learned_addr learned my addr 172.21.15.107:0/3724082632 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:05.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.470+0000 7f69d27fc640 10 client.?.objecter ms_handle_connect 0x7f69dc0a9470 2024-03-25T19:46:05.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.470+0000 7f69d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:05.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.470+0000 7f69d27fc640 10 client.?.objecter ms_handle_connect 0x7f69dc0b4360 2024-03-25T19:46:05.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.470+0000 7f69d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.477+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417208780 0 0) 0x7f69dc0b0920 con 0x7f69dc0b4360 2024-03-25T19:46:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.477+0000 7f69d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.477+0000 7f69d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.477+0000 7f69d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.477+0000 7f69d27fc640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:46:05.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.477+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c0004eb0 con 0x7f69dc0b4360 2024-03-25T19:46:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.478+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565179557 0 0) 0x7f69dc148370 con 0x7f69dc0a9470 2024-03-25T19:46:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.478+0000 7f69d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.478+0000 7f69d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.478+0000 7f69d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:05.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.478+0000 7f69d27fc640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:46:05.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.478+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69dc0b0920 con 0x7f69dc0a9470 2024-03-25T19:46:05.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.481+0000 7f69d27fc640 10 client.?.objecter ms_handle_connect 0x7f69dc0b10d0 2024-03-25T19:46:05.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.481+0000 7f69d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:05.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.496+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496451410 0 0) 0x7f69dc1485c0 con 0x7f69dc0b10d0 2024-03-25T19:46:05.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.496+0000 7f69d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:05.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.496+0000 7f69d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:05.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.496+0000 7f69d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:05.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.496+0000 7f69d27fc640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:46:05.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.496+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69dc148370 con 0x7f69dc0b10d0 2024-03-25T19:46:05.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.497+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2600584886 0 0) 0x7f69c0004eb0 con 0x7f69dc0b4360 2024-03-25T19:46:05.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.497+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69dc1485c0 con 0x7f69dc0b4360 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.497+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f69d4001c70 con 0x7f69dc0b4360 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.497+0000 7f69d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.497+0000 7f69d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.497+0000 7f69d27fc640 10 monclient(hunting): dump: 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:05.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.497+0000 7f69d27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:05.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.500+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 880568200 0 0) 0x7f69dc0b0920 con 0x7f69dc0a9470 2024-03-25T19:46:05.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.500+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69c0004eb0 con 0x7f69dc0a9470 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.500+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f69d8001950 con 0x7f69dc0a9470 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.500+0000 7f69d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.500+0000 7f69d27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.500+0000 7f69d27fc640 10 monclient(hunting): dump: 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:05.500 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:05.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3919443587 0 0) 0x7f69dc1485c0 con 0x7f69dc0b4360 2024-03-25T19:46:05.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 >> v1:172.21.15.107:6790/0 conn(0x7f69dc0b10d0 legacy=0x7f69dc1d43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 >> v1:172.21.15.107:6789/0 conn(0x7f69dc0a9470 legacy=0x7f69dc14e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69d27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:05.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69d27fc640 1 monclient: found mon.b 2024-03-25T19:46:05.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:05.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69d27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:05.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc1d7310 con 0x7f69dc0b4360 2024-03-25T19:46:05.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69d27fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:05.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69d27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:05.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:05.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69e4055640 5 monclient: authenticate success, global_id 4675 2024-03-25T19:46:05.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69e4055640 10 monclient: _renew_subs 2024-03-25T19:46:05.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69e4055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:05.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69e4055640 1 -- 172.21.15.107:0/3724082632 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc1d7560 con 0x7f69dc0b4360 2024-03-25T19:46:05.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69e4055640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:05.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69e4055640 10 monclient: _renew_subs 2024-03-25T19:46:05.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69e4055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:05.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69e4055640 1 -- 172.21.15.107:0/3724082632 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc14e930 con 0x7f69dc0b4360 2024-03-25T19:46:05.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.504+0000 7f69e4055640 1 librados: init done 2024-03-25T19:46:05.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.505+0000 7f69e4055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:05.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.505+0000 7f69e4055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:05.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.505+0000 7f69e4055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:05.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.505+0000 7f69e4055640 1 -- 172.21.15.107:0/3724082632 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69dc0b0770 con 0x7f69dc0b4360 2024-03-25T19:46:05.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.801+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69d4001dd0 con 0x7f69dc0b4360 2024-03-25T19:46:05.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.801+0000 7f69d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:05.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.801+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f69d4003b30 con 0x7f69dc0b4360 2024-03-25T19:46:05.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.801+0000 7f69d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:05.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.801+0000 7f69d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:05.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.801+0000 7f69d27fc640 10 monclient: dump: 2024-03-25T19:46:05.801 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:05.801 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:05.802 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:05.802 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:05.802 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:05.802 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:05.802 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:05.802 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:05.802 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:05.802 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:05.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.801+0000 7f69d3fff640 4 set_mon_vals no callback set 2024-03-25T19:46:05.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.802+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f69d4003d50 con 0x7f69dc0b4360 2024-03-25T19:46:05.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.803+0000 7f69e1df1640 10 monclient: get_auth_request con 0x7f69c006df70 auth_method 0 2024-03-25T19:46:05.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.803+0000 7f69d27fc640 10 client.4675.objecter ms_handle_connect 0x7f69c006df70 2024-03-25T19:46:05.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.810+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3155620770 0 0) 0x7f69d4054ab0 con 0x7f69dc0b4360 2024-03-25T19:46:05.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.810+0000 7f69d27fc640 10 client.4675.objecter ms_dispatch 0x7f69dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:05.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.810+0000 7f69d27fc640 3 client.4675.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:05.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.810+0000 7f69d27fc640 3 client.4675.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:05.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.811+0000 7f69d27fc640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T19:46:05.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.811+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69d40505c0 con 0x7f69dc0b4360 2024-03-25T19:46:05.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.811+0000 7f69d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:05.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:05.811+0000 7f69d27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:06.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.060+0000 7f69e4055640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:06.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.060+0000 7f69e4055640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:06.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.060+0000 7f69e4055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:06.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.060+0000 7f69e4055640 1 -- 172.21.15.107:0/3724082632 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69dc0627b0 con 0x7f69dc0b4360 2024-03-25T19:46:06.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.099+0000 7f69d27fc640 1 -- 172.21.15.107:0/3724082632 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2864573051) 0x7f69d40524f0 con 0x7f69dc0b4360 2024-03-25T19:46:06.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.099+0000 7f69d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:06.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.099+0000 7f69d27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:06.099 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:46:06.099 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:45:51.343698+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:51.343698+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:06.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 10 librados: watch_flush enter 2024-03-25T19:46:06.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 10 librados: watch_flush exit 2024-03-25T19:46:06.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T19:46:06.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 10 client.4675.objecter successfully canceled tick 2024-03-25T19:46:06.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 1 -- 172.21.15.107:0/3724082632 >> v1:172.21.15.195:6800/2984408421 conn(0x7f69c006df70 legacy=0x7f69c0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:06.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 10 monclient: shutdown 2024-03-25T19:46:06.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:06.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 1 -- 172.21.15.107:0/3724082632 >> v1:172.21.15.195:6789/0 conn(0x7f69dc0b4360 legacy=0x7f69dc1e2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:06.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 1 -- 172.21.15.107:0/3724082632 shutdown_connections 2024-03-25T19:46:06.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 1 -- 172.21.15.107:0/3724082632 >> 172.21.15.107:0/3724082632 conn(0x7f69dc0617a0 msgr2=0x7f69dc0b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:06.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 1 -- 172.21.15.107:0/3724082632 shutdown_connections 2024-03-25T19:46:06.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 1 -- 172.21.15.107:0/3724082632 wait complete. 2024-03-25T19:46:06.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:06.101+0000 7f69e4055640 1 librados: shutdown 2024-03-25T19:46:06.114 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:11.520 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:10:41.836 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:11.533 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:10:41.849 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:11.571 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:10:41.887 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:11.577 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:10:41.893 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:11.583 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:10:41.900 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:11.585 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:10:41.872 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:11.595 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:10:41.882 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:11.620 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:10:41.937 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:11.628 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:10:41.915 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:12.921 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:13.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.040+0000 7ff2714d6640 10 monclient: get_monmap_and_config 2024-03-25T19:46:13.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.040+0000 7ff2714d6640 10 monclient: build_initial_monmap 2024-03-25T19:46:13.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.040+0000 7ff2714d6640 10 monclient: monmap: 2024-03-25T19:46:13.040 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:13.040 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:13.040 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:46:13.041374+0000 2024-03-25T19:46:13.040 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:46:13.041374+0000 2024-03-25T19:46:13.040 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:13.040 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:13.040 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:13.041 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:46:13.041 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:46:13.041 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:13.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 1 Processor -- start 2024-03-25T19:46:13.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 1 -- start start 2024-03-25T19:46:13.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 10 monclient: init 2024-03-25T19:46:13.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:13.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:13.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff26c0b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:13.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff26c0b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:13.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff26c14be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:13.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c0af940 con 0x7ff26c0b4d60 2024-03-25T19:46:13.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c0b0a70 con 0x7ff26c0b1730 2024-03-25T19:46:13.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c0b0d50 con 0x7ff26c14be90 2024-03-25T19:46:13.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:13.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.041+0000 7ff2714d6640 10 monclient(hunting): authenticate will time out at 5165.382548s 2024-03-25T19:46:13.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.047+0000 7ff26a7fc640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff26c0b1730 0x7ff26c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:39950/0 (socket says 172.21.15.107:39950) 2024-03-25T19:46:13.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.047+0000 7ff26a7fc640 1 -- 172.21.15.107:0/3412418115 learned_addr learned my addr 172.21.15.107:0/3412418115 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:13.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.058+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161861673 0 0) 0x7ff26c0af940 con 0x7ff26c0b4d60 2024-03-25T19:46:13.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.058+0000 7ff269ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:13.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.058+0000 7ff269ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:13.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.058+0000 7ff269ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:13.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.058+0000 7ff269ffb640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:46:13.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.059+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff254003520 con 0x7ff26c0b4d60 2024-03-25T19:46:13.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.064+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952160674 0 0) 0x7ff26c0b0a70 con 0x7ff26c0b1730 2024-03-25T19:46:13.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.064+0000 7ff269ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:13.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.064+0000 7ff269ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:13.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.064+0000 7ff269ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:13.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.064+0000 7ff269ffb640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:46:13.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.064+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c0af940 con 0x7ff26c0b1730 2024-03-25T19:46:13.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.071+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415987608 0 0) 0x7ff26c0b0d50 con 0x7ff26c14be90 2024-03-25T19:46:13.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.071+0000 7ff269ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:13.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.071+0000 7ff269ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:13.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.071+0000 7ff269ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:13.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.071+0000 7ff269ffb640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:46:13.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.071+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c0b0a70 con 0x7ff26c14be90 2024-03-25T19:46:13.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 618714535 0 0) 0x7ff254003520 con 0x7ff26c0b4d60 2024-03-25T19:46:13.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 >> v1:172.21.15.107:6790/0 conn(0x7ff26c14be90 legacy=0x7ff26c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 >> v1:172.21.15.195:6789/0 conn(0x7ff26c0b1730 legacy=0x7ff26c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 1 monclient: found mon.noname-a 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c0b1030 con 0x7ff26c0b4d60 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff25c001690 con 0x7ff26c0b4d60 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff269ffb640 10 monclient: dump: 2024-03-25T19:46:13.076 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:13.077 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:13.077 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.077 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.077 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:13.077 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:13.077 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:13.077 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:13.077 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:13.077 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:13.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff2714d6640 5 monclient: authenticate success, global_id 4686 2024-03-25T19:46:13.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.075+0000 7ff2714d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:13.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.088+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff25c0017f0 con 0x7ff26c0b4d60 2024-03-25T19:46:13.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.088+0000 7ff269ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:13.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.088+0000 7ff269ffb640 1 -- 172.21.15.107:0/3412418115 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff25c001c00 con 0x7ff26c0b4d60 2024-03-25T19:46:13.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.088+0000 7ff2714d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:13.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.088+0000 7ff2714d6640 4 set_mon_vals no callback set 2024-03-25T19:46:13.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.088+0000 7ff269ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:13.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.088+0000 7ff269ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:13.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.088+0000 7ff269ffb640 10 monclient: dump: 2024-03-25T19:46:13.089 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:13.089 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:13.089 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.089 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.089 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:13.089 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:13.089 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:13.089 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:13.089 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:13.089 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.089+0000 7ff2714d6640 10 monclient: shutdown 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.089+0000 7ff2714d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.089+0000 7ff2714d6640 1 -- 172.21.15.107:0/3412418115 >> v1:172.21.15.107:6789/0 conn(0x7ff26c0b4d60 legacy=0x7ff26c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.089+0000 7ff2714d6640 1 -- 172.21.15.107:0/3412418115 shutdown_connections 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.089+0000 7ff2714d6640 1 -- 172.21.15.107:0/3412418115 >> 172.21.15.107:0/3412418115 conn(0x7ff26c0617a0 msgr2=0x7ff26c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.089+0000 7ff2714d6640 1 -- 172.21.15.107:0/3412418115 shutdown_connections 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.089+0000 7ff2714d6640 1 -- 172.21.15.107:0/3412418115 wait complete. 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.090+0000 7ff2714d6640 10 monclient: build_initial_monmap 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.090+0000 7ff2714d6640 10 monclient: monmap: 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:46:13.091 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.091+0000 7ff2714d6640 1 librados: starting msgr at 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.091+0000 7ff2714d6640 1 librados: starting objecter 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.091+0000 7ff2714d6640 1 Processor -- start 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.091+0000 7ff2714d6640 1 -- start start 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.091+0000 7ff2714d6640 1 librados: setting wanted keys 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.091+0000 7ff2714d6640 1 librados: calling monclient init 2024-03-25T19:46:13.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.091+0000 7ff2714d6640 10 monclient: init 2024-03-25T19:46:13.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.091+0000 7ff2714d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:13.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.091+0000 7ff2714d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:13.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.091+0000 7ff2714d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff26c0b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:13.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.092+0000 7ff2714d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff26c0b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:13.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.092+0000 7ff2714d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff26c14be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:13.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.092+0000 7ff2714d6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c1368f0 con 0x7ff26c0b4d60 2024-03-25T19:46:13.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.092+0000 7ff2714d6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c137750 con 0x7ff26c0b1730 2024-03-25T19:46:13.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.092+0000 7ff2714d6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c1363d0 con 0x7ff26c14be90 2024-03-25T19:46:13.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.092+0000 7ff2714d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:13.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.092+0000 7ff2714d6640 10 monclient(hunting): authenticate will time out at 5165.432937s 2024-03-25T19:46:13.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.107+0000 7ff26b7fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff26c14be90 0x7ff26c1d7830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:58956/0 (socket says 172.21.15.107:58956) 2024-03-25T19:46:13.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.107+0000 7ff26b7fe640 1 -- 172.21.15.107:0/2289312926 learned_addr learned my addr 172.21.15.107:0/2289312926 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:13.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.108+0000 7ff2537fe640 10 client.?.objecter ms_handle_connect 0x7ff26c0b1730 2024-03-25T19:46:13.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.108+0000 7ff2537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:13.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.109+0000 7ff2537fe640 10 client.?.objecter ms_handle_connect 0x7ff26c14be90 2024-03-25T19:46:13.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.109+0000 7ff2537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:13.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.109+0000 7ff2537fe640 10 client.?.objecter ms_handle_connect 0x7ff26c0b4d60 2024-03-25T19:46:13.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.109+0000 7ff2537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.118+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748529692 0 0) 0x7ff26c137750 con 0x7ff26c0b1730 2024-03-25T19:46:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.118+0000 7ff2537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.118+0000 7ff2537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.118+0000 7ff2537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.118+0000 7ff2537fe640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:46:13.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.118+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff258004eb0 con 0x7ff26c0b1730 2024-03-25T19:46:13.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.124+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739616557 0 0) 0x7ff26c1368f0 con 0x7ff26c0b4d60 2024-03-25T19:46:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.124+0000 7ff2537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.124+0000 7ff2537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.124+0000 7ff2537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.124+0000 7ff2537fe640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:46:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.124+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c137750 con 0x7ff26c0b4d60 2024-03-25T19:46:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.126+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478886451 0 0) 0x7ff26c1363d0 con 0x7ff26c14be90 2024-03-25T19:46:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.126+0000 7ff2537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.126+0000 7ff2537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:13.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.126+0000 7ff2537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:13.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.126+0000 7ff2537fe640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:46:13.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.126+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c1368f0 con 0x7ff26c14be90 2024-03-25T19:46:13.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.135+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2260757766 0 0) 0x7ff26c137750 con 0x7ff26c0b4d60 2024-03-25T19:46:13.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.135+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff26c1363d0 con 0x7ff26c0b4d60 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.135+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff260001970 con 0x7ff26c0b4d60 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.135+0000 7ff2537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.136+0000 7ff2537fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.136+0000 7ff2537fe640 10 monclient(hunting): dump: 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:13.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.136+0000 7ff2537fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:13.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.139+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 924314758 0 0) 0x7ff258004eb0 con 0x7ff26c0b1730 2024-03-25T19:46:13.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.139+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff26c137750 con 0x7ff26c0b1730 2024-03-25T19:46:13.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.139+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff25c001b00 con 0x7ff26c0b1730 2024-03-25T19:46:13.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.139+0000 7ff2537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:13.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.139+0000 7ff2537fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:13.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.139+0000 7ff2537fe640 10 monclient(hunting): dump: 2024-03-25T19:46:13.139 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:13.140 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:13.140 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.140 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.140 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:13.140 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:13.140 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:13.140 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:13.140 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:13.140 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.144+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1517465308 0 0) 0x7ff26c1368f0 con 0x7ff26c14be90 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.144+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff258004eb0 con 0x7ff26c14be90 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.144+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff264001bd0 con 0x7ff26c14be90 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.144+0000 7ff2537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.144+0000 7ff2537fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.144+0000 7ff2537fe640 10 monclient(hunting): dump: 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:13.144 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:13.145 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:13.145 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:13.145 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:13.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 423954680 0 0) 0x7ff26c137750 con 0x7ff26c0b1730 2024-03-25T19:46:13.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 >> v1:172.21.15.107:6790/0 conn(0x7ff26c14be90 legacy=0x7ff26c1d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:13.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 >> v1:172.21.15.107:6789/0 conn(0x7ff26c0b4d60 legacy=0x7ff26c12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2537fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2537fe640 1 monclient: found mon.b 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2537fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c130470 con 0x7ff26c0b1730 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2537fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2537fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2714d6640 5 monclient: authenticate success, global_id 4681 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2714d6640 10 monclient: _renew_subs 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2714d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:13.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2714d6640 1 -- 172.21.15.107:0/2289312926 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26c130720 con 0x7ff26c0b1730 2024-03-25T19:46:13.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2714d6640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:13.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2714d6640 10 monclient: _renew_subs 2024-03-25T19:46:13.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2714d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:13.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2714d6640 1 -- 172.21.15.107:0/2289312926 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26c1322f0 con 0x7ff26c0b1730 2024-03-25T19:46:13.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.153+0000 7ff2714d6640 1 librados: init done 2024-03-25T19:46:13.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.155+0000 7ff2714d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:13.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.155+0000 7ff2714d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:13.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.155+0000 7ff2714d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:13.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.155+0000 7ff2714d6640 1 -- 172.21.15.107:0/2289312926 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26c136210 con 0x7ff26c0b1730 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.443+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff25c0038a0 con 0x7ff26c0b1730 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.443+0000 7ff2537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.443+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7ff25c003bb0 con 0x7ff26c0b1730 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.443+0000 7ff2537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.443+0000 7ff2537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.443+0000 7ff2537fe640 10 monclient: dump: 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:13.443 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:13.444 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:13.444 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:13.444 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:13.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.443+0000 7ff2697fa640 4 set_mon_vals no callback set 2024-03-25T19:46:13.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.443+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7ff25c019b80 con 0x7ff26c0b1730 2024-03-25T19:46:13.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.444+0000 7ff26a7fc640 10 monclient: get_auth_request con 0x7ff25806e160 auth_method 0 2024-03-25T19:46:13.444 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.445+0000 7ff2537fe640 10 client.4681.objecter ms_handle_connect 0x7ff25806e160 2024-03-25T19:46:13.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.450+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3155620770 0 0) 0x7ff25c053680 con 0x7ff26c0b1730 2024-03-25T19:46:13.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.450+0000 7ff2537fe640 10 client.4681.objecter ms_dispatch 0x7ff26c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:13.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.450+0000 7ff2537fe640 3 client.4681.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:13.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.450+0000 7ff2537fe640 3 client.4681.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:13.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.450+0000 7ff2537fe640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T19:46:13.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.450+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff25c04f190 con 0x7ff26c0b1730 2024-03-25T19:46:13.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.450+0000 7ff2537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:13.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.450+0000 7ff2537fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:13.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.699+0000 7ff2714d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:13.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.699+0000 7ff2714d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:13.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.699+0000 7ff2714d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:13.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.699+0000 7ff2714d6640 1 -- 172.21.15.107:0/2289312926 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff26c0b2270 con 0x7ff26c0b1730 2024-03-25T19:46:13.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.727+0000 7ff2537fe640 1 -- 172.21.15.107:0/2289312926 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2864573051) 0x7ff25c0510c0 con 0x7ff26c0b1730 2024-03-25T19:46:13.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.727+0000 7ff2537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:13.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.727+0000 7ff2537fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:13.727 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:46:13.727 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:45:51.343698+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:51.343698+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:13.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 10 librados: watch_flush enter 2024-03-25T19:46:13.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 10 librados: watch_flush exit 2024-03-25T19:46:13.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T19:46:13.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 10 client.4681.objecter successfully canceled tick 2024-03-25T19:46:13.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 1 -- 172.21.15.107:0/2289312926 >> v1:172.21.15.195:6800/2984408421 conn(0x7ff25806e160 legacy=0x7ff258070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:13.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 10 monclient: shutdown 2024-03-25T19:46:13.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:13.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 1 -- 172.21.15.107:0/2289312926 >> v1:172.21.15.195:6789/0 conn(0x7ff26c0b1730 legacy=0x7ff26c135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:13.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 1 -- 172.21.15.107:0/2289312926 shutdown_connections 2024-03-25T19:46:13.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 1 -- 172.21.15.107:0/2289312926 >> 172.21.15.107:0/2289312926 conn(0x7ff26c0617a0 msgr2=0x7ff26c138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:13.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 1 -- 172.21.15.107:0/2289312926 shutdown_connections 2024-03-25T19:46:13.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 1 -- 172.21.15.107:0/2289312926 wait complete. 2024-03-25T19:46:13.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:13.729+0000 7ff2714d6640 1 librados: shutdown 2024-03-25T19:46:13.743 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:20.548 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:20.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.664+0000 7fb96d6c9640 10 monclient: get_monmap_and_config 2024-03-25T19:46:20.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.664+0000 7fb96d6c9640 10 monclient: build_initial_monmap 2024-03-25T19:46:20.664 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.664+0000 7fb96d6c9640 10 monclient: monmap: 2024-03-25T19:46:20.664 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:46:20.665253+0000 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:46:20.665253+0000 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.664+0000 7fb96d6c9640 1 Processor -- start 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 1 -- start start 2024-03-25T19:46:20.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 10 monclient: init 2024-03-25T19:46:20.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:20.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:20.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 10 monclient(hunting): picked mon.noname-c con 0x7fb9680a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:20.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 10 monclient(hunting): picked mon.noname-a con 0x7fb96813f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:20.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 10 monclient(hunting): picked mon.noname-b con 0x7fb9681430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:20.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb968142520 con 0x7fb96813f710 2024-03-25T19:46:20.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9680b28a0 con 0x7fb9680a98f0 2024-03-25T19:46:20.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9680b2a30 con 0x7fb9681430c0 2024-03-25T19:46:20.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:20.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.665+0000 7fb96d6c9640 10 monclient(hunting): authenticate will time out at 5173.006656s 2024-03-25T19:46:20.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.670+0000 7fb9667fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb96813f710 0x7fb9681429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37328/0 (socket says 172.21.15.107:37328) 2024-03-25T19:46:20.670 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.670+0000 7fb9667fc640 1 -- 172.21.15.107:0/617302474 learned_addr learned my addr 172.21.15.107:0/617302474 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:20.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.674+0000 7fb965ffb640 1 -- 172.21.15.107:0/617302474 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763937521 0 0) 0x7fb968142520 con 0x7fb96813f710 2024-03-25T19:46:20.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.674+0000 7fb965ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:20.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.674+0000 7fb965ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:20.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.674+0000 7fb965ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:20.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.674+0000 7fb965ffb640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:46:20.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.675+0000 7fb965ffb640 1 -- 172.21.15.107:0/617302474 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb950003520 con 0x7fb96813f710 2024-03-25T19:46:20.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 1 -- 172.21.15.107:0/617302474 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4008792899 0 0) 0x7fb950003520 con 0x7fb96813f710 2024-03-25T19:46:20.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 1 -- 172.21.15.107:0/617302474 >> v1:172.21.15.107:6790/0 conn(0x7fb9681430c0 legacy=0x7fb96814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 1 -- 172.21.15.107:0/617302474 >> v1:172.21.15.195:6789/0 conn(0x7fb9680a98f0 legacy=0x7fb9680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 1 monclient: found mon.noname-a 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 1 -- 172.21.15.107:0/617302474 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9680b44c0 con 0x7fb96813f710 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 1 -- 172.21.15.107:0/617302474 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb95c001690 con 0x7fb96813f710 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:20.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb965ffb640 10 monclient: dump: 2024-03-25T19:46:20.682 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:20.682 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:20.682 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:20.682 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:20.682 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:20.682 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:20.682 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:20.682 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:20.682 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:20.682 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:20.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb96d6c9640 5 monclient: authenticate success, global_id 4692 2024-03-25T19:46:20.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.680+0000 7fb96d6c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:20.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.693+0000 7fb965ffb640 1 -- 172.21.15.107:0/617302474 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb95c0017f0 con 0x7fb96813f710 2024-03-25T19:46:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.693+0000 7fb965ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.693+0000 7fb965ffb640 1 -- 172.21.15.107:0/617302474 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb95c001c40 con 0x7fb96813f710 2024-03-25T19:46:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.693+0000 7fb965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.693+0000 7fb965ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:20.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.693+0000 7fb965ffb640 10 monclient: dump: 2024-03-25T19:46:20.693 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:20.693 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:20.693 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:20.693 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:20.693 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:20.693 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:20.694 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:20.694 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:20.694 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:20.694 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:20.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.693+0000 7fb96d6c9640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:20.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.693+0000 7fb96d6c9640 4 set_mon_vals no callback set 2024-03-25T19:46:20.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.694+0000 7fb96d6c9640 10 monclient: shutdown 2024-03-25T19:46:20.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.694+0000 7fb96d6c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:20.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.694+0000 7fb96d6c9640 1 -- 172.21.15.107:0/617302474 >> v1:172.21.15.107:6789/0 conn(0x7fb96813f710 legacy=0x7fb9681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.694+0000 7fb96d6c9640 1 -- 172.21.15.107:0/617302474 shutdown_connections 2024-03-25T19:46:20.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.694+0000 7fb96d6c9640 1 -- 172.21.15.107:0/617302474 >> 172.21.15.107:0/617302474 conn(0x7fb9680b7ea0 msgr2=0x7fb9680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:20.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.694+0000 7fb96d6c9640 1 -- 172.21.15.107:0/617302474 shutdown_connections 2024-03-25T19:46:20.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.694+0000 7fb96d6c9640 1 -- 172.21.15.107:0/617302474 wait complete. 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.695+0000 7fb96d6c9640 10 monclient: build_initial_monmap 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.695+0000 7fb96d6c9640 10 monclient: monmap: 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:20.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.695+0000 7fb96d6c9640 1 librados: starting msgr at 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.695+0000 7fb96d6c9640 1 librados: starting objecter 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.695+0000 7fb96d6c9640 1 Processor -- start 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 1 -- start start 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 1 librados: setting wanted keys 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 1 librados: calling monclient init 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 10 monclient: init 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 10 monclient(hunting): picked mon.noname-a con 0x7fb9680a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 10 monclient(hunting): picked mon.noname-c con 0x7fb96813f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 10 monclient(hunting): picked mon.noname-b con 0x7fb9681430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9680abd70 con 0x7fb9680a98f0 2024-03-25T19:46:20.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9681d7710 con 0x7fb9681430c0 2024-03-25T19:46:20.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9681d8340 con 0x7fb96813f710 2024-03-25T19:46:20.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:20.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.696+0000 7fb96d6c9640 10 monclient(hunting): authenticate will time out at 5173.037426s 2024-03-25T19:46:20.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.703+0000 7fb9677fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb9681430c0 0x7fb9681d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:44276/0 (socket says 172.21.15.107:44276) 2024-03-25T19:46:20.703 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.703+0000 7fb9677fe640 1 -- 172.21.15.107:0/4205016138 learned_addr learned my addr 172.21.15.107:0/4205016138 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:20.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.706+0000 7fb94f7fe640 10 client.?.objecter ms_handle_connect 0x7fb9681430c0 2024-03-25T19:46:20.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.706+0000 7fb94f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:20.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.710+0000 7fb94f7fe640 10 client.?.objecter ms_handle_connect 0x7fb96813f710 2024-03-25T19:46:20.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.710+0000 7fb94f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:20.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.711+0000 7fb94f7fe640 10 client.?.objecter ms_handle_connect 0x7fb9680a98f0 2024-03-25T19:46:20.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.711+0000 7fb94f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.722+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871863477 0 0) 0x7fb9680abd70 con 0x7fb9680a98f0 2024-03-25T19:46:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.722+0000 7fb94f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.722+0000 7fb94f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.722+0000 7fb94f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.722+0000 7fb94f7fe640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:46:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.722+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb954004eb0 con 0x7fb9680a98f0 2024-03-25T19:46:20.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.732+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361801671 0 0) 0x7fb9681d7710 con 0x7fb9681430c0 2024-03-25T19:46:20.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.732+0000 7fb94f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:20.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.732+0000 7fb94f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:20.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.732+0000 7fb94f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:20.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.732+0000 7fb94f7fe640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:46:20.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.732+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9680abd70 con 0x7fb9681430c0 2024-03-25T19:46:20.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.733+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504637628 0 0) 0x7fb9681d8340 con 0x7fb96813f710 2024-03-25T19:46:20.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.733+0000 7fb94f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:20.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.733+0000 7fb94f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:20.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.733+0000 7fb94f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:20.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.733+0000 7fb94f7fe640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:46:20.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.733+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9681d7710 con 0x7fb96813f710 2024-03-25T19:46:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.736+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4231003331 0 0) 0x7fb954004eb0 con 0x7fb9680a98f0 2024-03-25T19:46:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.736+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9681d8340 con 0x7fb9680a98f0 2024-03-25T19:46:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.736+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb958001930 con 0x7fb9680a98f0 2024-03-25T19:46:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.736+0000 7fb94f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.736+0000 7fb94f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.736+0000 7fb94f7fe640 10 monclient(hunting): dump: 2024-03-25T19:46:20.736 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:20.736 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:20.736 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:20.737 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:20.737 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:20.737 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:20.737 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:20.737 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:20.737 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:20.737 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.736+0000 7fb94f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:20.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.745+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2677337342 0 0) 0x7fb9681d8340 con 0x7fb9680a98f0 2024-03-25T19:46:20.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 >> v1:172.21.15.107:6790/0 conn(0x7fb96813f710 legacy=0x7fb9681d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 >> v1:172.21.15.195:6789/0 conn(0x7fb9681430c0 legacy=0x7fb9681d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb94f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:20.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb94f7fe640 1 monclient: found mon.a 2024-03-25T19:46:20.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb94f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:20.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb94f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:46:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9681d84d0 con 0x7fb9680a98f0 2024-03-25T19:46:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb94f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb94f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb94f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb96d6c9640 5 monclient: authenticate success, global_id 4695 2024-03-25T19:46:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb96d6c9640 10 monclient: _renew_subs 2024-03-25T19:46:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb96d6c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:46:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb96d6c9640 1 -- 172.21.15.107:0/4205016138 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9681d41e0 con 0x7fb9680a98f0 2024-03-25T19:46:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb96d6c9640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb96d6c9640 10 monclient: _renew_subs 2024-03-25T19:46:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb96d6c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:46:20.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb96d6c9640 1 -- 172.21.15.107:0/4205016138 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9681d5150 con 0x7fb9680a98f0 2024-03-25T19:46:20.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.746+0000 7fb96d6c9640 1 librados: init done 2024-03-25T19:46:20.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.747+0000 7fb96d6c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:20.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.747+0000 7fb96d6c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:20.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.747+0000 7fb96d6c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:46:20.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:20.747+0000 7fb96d6c9640 1 -- 172.21.15.107:0/4205016138 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9680627b0 con 0x7fb9680a98f0 2024-03-25T19:46:21.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.038+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb958003610 con 0x7fb9680a98f0 2024-03-25T19:46:21.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.038+0000 7fb94f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:21.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.038+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7fb958003a20 con 0x7fb9680a98f0 2024-03-25T19:46:21.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.038+0000 7fb94f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:21.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.038+0000 7fb94f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:21.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.038+0000 7fb94f7fe640 10 monclient: dump: 2024-03-25T19:46:21.038 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:21.038 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:21.038 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:21.038 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:21.039 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:21.039 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:21.039 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:21.039 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:21.039 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:21.039 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:21.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.038+0000 7fb9657fa640 4 set_mon_vals no callback set 2024-03-25T19:46:21.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.039+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7fb958003c40 con 0x7fb9680a98f0 2024-03-25T19:46:21.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.039+0000 7fb9667fc640 10 monclient: get_auth_request con 0x7fb95406de20 auth_method 0 2024-03-25T19:46:21.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.040+0000 7fb94f7fe640 10 client.4695.objecter ms_handle_connect 0x7fb95406de20 2024-03-25T19:46:21.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.045+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3155620770 0 0) 0x7fb9580535b0 con 0x7fb9680a98f0 2024-03-25T19:46:21.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.045+0000 7fb94f7fe640 10 client.4695.objecter ms_dispatch 0x7fb968000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:21.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.045+0000 7fb94f7fe640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:21.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.045+0000 7fb94f7fe640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:21.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.045+0000 7fb94f7fe640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T19:46:21.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.045+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb95804f0c0 con 0x7fb9680a98f0 2024-03-25T19:46:21.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.045+0000 7fb94f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:21.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.045+0000 7fb94f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:21.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.298+0000 7fb96d6c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:21.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.298+0000 7fb96d6c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:21.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.298+0000 7fb96d6c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:46:21.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.298+0000 7fb96d6c9640 1 -- 172.21.15.107:0/4205016138 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9681d78a0 con 0x7fb9680a98f0 2024-03-25T19:46:21.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.418+0000 7fb94f7fe640 1 -- 172.21.15.107:0/4205016138 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2864573051) 0x7fb958050ff0 con 0x7fb9680a98f0 2024-03-25T19:46:21.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.418+0000 7fb94f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:21.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.418+0000 7fb94f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:21.419 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:46:21.419 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:45:51.343698+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:51.343698+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:21.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.420+0000 7fb96d6c9640 10 librados: watch_flush enter 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.420+0000 7fb96d6c9640 10 librados: watch_flush exit 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.420+0000 7fb96d6c9640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.420+0000 7fb96d6c9640 10 client.4695.objecter successfully canceled tick 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.421+0000 7fb96d6c9640 1 -- 172.21.15.107:0/4205016138 >> v1:172.21.15.195:6800/2984408421 conn(0x7fb95406de20 legacy=0x7fb9540702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.421+0000 7fb96d6c9640 10 monclient: shutdown 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.421+0000 7fb96d6c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.421+0000 7fb96d6c9640 1 -- 172.21.15.107:0/4205016138 >> v1:172.21.15.107:6789/0 conn(0x7fb9680a98f0 legacy=0x7fb968137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.421+0000 7fb96d6c9640 1 -- 172.21.15.107:0/4205016138 shutdown_connections 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.421+0000 7fb96d6c9640 1 -- 172.21.15.107:0/4205016138 >> 172.21.15.107:0/4205016138 conn(0x7fb9680b7ea0 msgr2=0x7fb968074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.421+0000 7fb96d6c9640 1 -- 172.21.15.107:0/4205016138 shutdown_connections 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.421+0000 7fb96d6c9640 1 -- 172.21.15.107:0/4205016138 wait complete. 2024-03-25T19:46:21.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:21.421+0000 7fb96d6c9640 1 librados: shutdown 2024-03-25T19:46:21.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:28.239 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:28.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.346+0000 7f153fd3b640 10 monclient: get_monmap_and_config 2024-03-25T19:46:28.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.346+0000 7f153fd3b640 10 monclient: build_initial_monmap 2024-03-25T19:46:28.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.346+0000 7f153fd3b640 10 monclient: monmap: 2024-03-25T19:46:28.346 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:28.346 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:28.346 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:46:28.347338+0000 2024-03-25T19:46:28.346 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:46:28.347338+0000 2024-03-25T19:46:28.346 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:28.346 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:28.346 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:28.346 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:46:28.347 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:46:28.347 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:28.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.347+0000 7f153fd3b640 1 Processor -- start 2024-03-25T19:46:28.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.347+0000 7f153fd3b640 1 -- start start 2024-03-25T19:46:28.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.347+0000 7f153fd3b640 10 monclient: init 2024-03-25T19:46:28.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.348+0000 7f153fd3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:28.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.348+0000 7f153fd3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:28.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.348+0000 7f153fd3b640 10 monclient(hunting): picked mon.noname-a con 0x7f1538061830 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:28.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.348+0000 7f153fd3b640 10 monclient(hunting): picked mon.noname-c con 0x7f15380ae840 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:28.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.348+0000 7f153fd3b640 10 monclient(hunting): picked mon.noname-b con 0x7f153813fbd0 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:28.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.348+0000 7f153fd3b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15380ae1c0 con 0x7f1538061830 2024-03-25T19:46:28.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.348+0000 7f153fd3b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153806f7d0 con 0x7f15380ae840 2024-03-25T19:46:28.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.348+0000 7f153fd3b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153806fa40 con 0x7f153813fbd0 2024-03-25T19:46:28.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.348+0000 7f153fd3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:28.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.348+0000 7f153fd3b640 10 monclient(hunting): authenticate will time out at 5180.689230s 2024-03-25T19:46:28.355 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.355+0000 7f153d2d6640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f15380ae840 0x7f1538145810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:48958/0 (socket says 172.21.15.107:48958) 2024-03-25T19:46:28.355 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.355+0000 7f153d2d6640 1 -- 172.21.15.107:0/3019585073 learned_addr learned my addr 172.21.15.107:0/3019585073 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326365956 0 0) 0x7f15380ae1c0 con 0x7f1538061830 2024-03-25T19:46:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 10 monclient(hunting): _negotiate 2024-03-25T19:46:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:46:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1524003520 con 0x7f1538061830 2024-03-25T19:46:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904153930 0 0) 0x7f153806f7d0 con 0x7f15380ae840 2024-03-25T19:46:28.369 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 10 monclient(hunting): _negotiate 2024-03-25T19:46:28.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:28.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:28.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:46:28.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.369+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15380ae1c0 con 0x7f15380ae840 2024-03-25T19:46:28.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.376+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181595813 0 0) 0x7f153806fa40 con 0x7f153813fbd0 2024-03-25T19:46:28.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.376+0000 7f153cad5640 10 monclient(hunting): _negotiate 2024-03-25T19:46:28.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.376+0000 7f153cad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:28.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.376+0000 7f153cad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:28.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.376+0000 7f153cad5640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:46:28.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.376+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f153806f7d0 con 0x7f153813fbd0 2024-03-25T19:46:28.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2273483306 0 0) 0x7f1524003520 con 0x7f1538061830 2024-03-25T19:46:28.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 >> v1:172.21.15.107:6790/0 conn(0x7f153813fbd0 legacy=0x7f1538145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 >> v1:172.21.15.195:6789/0 conn(0x7f15380ae840 legacy=0x7f1538145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 10 monclient: _finish_hunting 0 2024-03-25T19:46:28.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 1 monclient: found mon.noname-a 2024-03-25T19:46:28.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1538073040 con 0x7f1538061830 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 10 monclient: _finish_auth 0 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 10 monclient: _check_auth_tickets 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1534001690 con 0x7f1538061830 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153cad5640 10 monclient: dump: 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:28.379 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:28.380 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:28.380 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:28.380 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:28.380 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:28.380 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:28.380 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:28.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153fd3b640 5 monclient: authenticate success, global_id 4698 2024-03-25T19:46:28.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.378+0000 7f153fd3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:28.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.399+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15340017f0 con 0x7f1538061830 2024-03-25T19:46:28.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.399+0000 7f153cad5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:28.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.399+0000 7f153cad5640 1 -- 172.21.15.107:0/3019585073 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1534001b80 con 0x7f1538061830 2024-03-25T19:46:28.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.399+0000 7f153cad5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:28.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.399+0000 7f153cad5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:28.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.399+0000 7f153cad5640 10 monclient: dump: 2024-03-25T19:46:28.399 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:28.399 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:28.399 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:28.400 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:28.400 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:28.400 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:28.400 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:28.400 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:28.400 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:28.400 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:28.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.399+0000 7f153fd3b640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:28.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.399+0000 7f153fd3b640 4 set_mon_vals no callback set 2024-03-25T19:46:28.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.400+0000 7f153fd3b640 10 monclient: shutdown 2024-03-25T19:46:28.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.400+0000 7f153fd3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:28.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.400+0000 7f153fd3b640 1 -- 172.21.15.107:0/3019585073 >> v1:172.21.15.107:6789/0 conn(0x7f1538061830 legacy=0x7f153813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.400+0000 7f153fd3b640 1 -- 172.21.15.107:0/3019585073 shutdown_connections 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.400+0000 7f153fd3b640 1 -- 172.21.15.107:0/3019585073 >> 172.21.15.107:0/3019585073 conn(0x7f15380b0ef0 msgr2=0x7f15380b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.400+0000 7f153fd3b640 1 -- 172.21.15.107:0/3019585073 shutdown_connections 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.400+0000 7f153fd3b640 1 -- 172.21.15.107:0/3019585073 wait complete. 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.401+0000 7f153fd3b640 10 monclient: build_initial_monmap 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.401+0000 7f153fd3b640 10 monclient: monmap: 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:46:28.403 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.401+0000 7f153fd3b640 1 librados: starting msgr at 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.401+0000 7f153fd3b640 1 librados: starting objecter 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.401+0000 7f153fd3b640 1 Processor -- start 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 1 -- start start 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 1 librados: setting wanted keys 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 1 librados: calling monclient init 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 10 monclient: init 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 10 monclient(hunting): picked mon.noname-b con 0x7f1538061830 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 10 monclient(hunting): picked mon.noname-c con 0x7f15380ae840 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 10 monclient(hunting): picked mon.noname-a con 0x7f153813fbd0 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153813bee0 con 0x7f153813fbd0 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153813c150 con 0x7f1538061830 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1538139dd0 con 0x7f15380ae840 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:28.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.402+0000 7f153fd3b640 10 monclient(hunting): authenticate will time out at 5180.743549s 2024-03-25T19:46:28.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.409+0000 7f153dad7640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1538061830 0x7f153813ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:48984/0 (socket says 172.21.15.107:48984) 2024-03-25T19:46:28.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.409+0000 7f153dad7640 1 -- 172.21.15.107:0/875526632 learned_addr learned my addr 172.21.15.107:0/875526632 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:28.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.411+0000 7f152e7fc640 10 client.?.objecter ms_handle_connect 0x7f1538061830 2024-03-25T19:46:28.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.411+0000 7f152e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:28.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.415+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006823301 0 0) 0x7f153813c150 con 0x7f1538061830 2024-03-25T19:46:28.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.415+0000 7f152e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:28.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.415+0000 7f152e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:28.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.415+0000 7f152e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:28.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.415+0000 7f152e7fc640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:46:28.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.415+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c004eb0 con 0x7f1538061830 2024-03-25T19:46:28.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.419+0000 7f152e7fc640 10 client.?.objecter ms_handle_connect 0x7f15380ae840 2024-03-25T19:46:28.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.419+0000 7f152e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:28.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.424+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388341788 0 0) 0x7f1538139dd0 con 0x7f15380ae840 2024-03-25T19:46:28.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.424+0000 7f152e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:28.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.424+0000 7f152e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:28.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.424+0000 7f152e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:28.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.424+0000 7f152e7fc640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:46:28.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.424+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f153813c150 con 0x7f15380ae840 2024-03-25T19:46:28.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.425+0000 7f152e7fc640 10 client.?.objecter ms_handle_connect 0x7f153813fbd0 2024-03-25T19:46:28.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.425+0000 7f152e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:28.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.434+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439580835 0 0) 0x7f153813bee0 con 0x7f153813fbd0 2024-03-25T19:46:28.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.434+0000 7f152e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:28.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.434+0000 7f152e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:28.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.434+0000 7f152e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:28.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.434+0000 7f152e7fc640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:46:28.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.434+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1538139dd0 con 0x7f153813fbd0 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.439+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 628326751 0 0) 0x7f151c004eb0 con 0x7f1538061830 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.439+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f153813bee0 con 0x7f1538061830 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.439+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1534001bc0 con 0x7f1538061830 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.439+0000 7f152e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.439+0000 7f152e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.439+0000 7f152e7fc640 10 monclient(hunting): dump: 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:28.439 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:28.440 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:28.440 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:28.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.439+0000 7f152e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.443+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2581255397 0 0) 0x7f1538139dd0 con 0x7f153813fbd0 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.443+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f151c004eb0 con 0x7f153813fbd0 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.443+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1530001c70 con 0x7f153813fbd0 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.443+0000 7f152e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.443+0000 7f152e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.443+0000 7f152e7fc640 10 monclient(hunting): dump: 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:28.443 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:28.444 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:28.444 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:28.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 233304589 0 0) 0x7f153813bee0 con 0x7f1538061830 2024-03-25T19:46:28.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 >> v1:172.21.15.107:6790/0 conn(0x7f15380ae840 legacy=0x7f153813ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 >> v1:172.21.15.107:6789/0 conn(0x7f153813fbd0 legacy=0x7f153813d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f152e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:28.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f152e7fc640 1 monclient: found mon.b 2024-03-25T19:46:28.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f152e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:28.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f152e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:28.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1538139f60 con 0x7f1538061830 2024-03-25T19:46:28.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f152e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:28.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f152e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:28.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f152e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:28.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f153fd3b640 5 monclient: authenticate success, global_id 4693 2024-03-25T19:46:28.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f153fd3b640 10 monclient: _renew_subs 2024-03-25T19:46:28.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f153fd3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:28.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.445+0000 7f153fd3b640 1 -- 172.21.15.107:0/875526632 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153813b280 con 0x7f1538061830 2024-03-25T19:46:28.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.446+0000 7f153fd3b640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:28.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.446+0000 7f153fd3b640 10 monclient: _renew_subs 2024-03-25T19:46:28.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.446+0000 7f153fd3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:28.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.446+0000 7f153fd3b640 1 -- 172.21.15.107:0/875526632 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15381e3740 con 0x7f1538061830 2024-03-25T19:46:28.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.446+0000 7f153fd3b640 1 librados: init done 2024-03-25T19:46:28.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.447+0000 7f153fd3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:28.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.447+0000 7f153fd3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:28.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.447+0000 7f153fd3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:28.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.447+0000 7f153fd3b640 1 -- 172.21.15.107:0/875526632 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15380627b0 con 0x7f1538061830 2024-03-25T19:46:28.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.722+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1534001210 con 0x7f1538061830 2024-03-25T19:46:28.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.722+0000 7f152e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:28.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.722+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1534001d80 con 0x7f1538061830 2024-03-25T19:46:28.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.722+0000 7f152e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:28.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.722+0000 7f152e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:28.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.722+0000 7f152e7fc640 10 monclient: dump: 2024-03-25T19:46:28.722 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:28.722 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:28.722 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:28.722 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:28.723 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:28.723 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:28.723 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:28.723 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:28.723 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:28.723 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:28.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.722+0000 7f152ffff640 4 set_mon_vals no callback set 2024-03-25T19:46:28.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.723+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f1534003b00 con 0x7f1538061830 2024-03-25T19:46:28.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.723+0000 7f153d2d6640 10 monclient: get_auth_request con 0x7f151c06ded0 auth_method 0 2024-03-25T19:46:28.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.724+0000 7f152e7fc640 10 client.4693.objecter ms_handle_connect 0x7f151c06ded0 2024-03-25T19:46:28.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.728+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3155620770 0 0) 0x7f15340547c0 con 0x7f1538061830 2024-03-25T19:46:28.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.728+0000 7f152e7fc640 10 client.4693.objecter ms_dispatch 0x7f1538000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:28.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.728+0000 7f152e7fc640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:28.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.728+0000 7f152e7fc640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:28.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.728+0000 7f152e7fc640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T19:46:28.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.729+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15340502d0 con 0x7f1538061830 2024-03-25T19:46:28.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.729+0000 7f152e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:28.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.729+0000 7f152e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:28.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.983+0000 7f153fd3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:28.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.983+0000 7f153fd3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:28.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.983+0000 7f153fd3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:28.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:28.983+0000 7f153fd3b640 1 -- 172.21.15.107:0/875526632 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1538068040 con 0x7f1538061830 2024-03-25T19:46:29.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.013+0000 7f152e7fc640 1 -- 172.21.15.107:0/875526632 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2864573051) 0x7f1534052200 con 0x7f1538061830 2024-03-25T19:46:29.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.013+0000 7f152e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:29.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.013+0000 7f152e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:29.013 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:46:29.013 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:45:51.343698+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:51.343698+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:29.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 10 librados: watch_flush enter 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 10 librados: watch_flush exit 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 10 client.4693.objecter successfully canceled tick 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 1 -- 172.21.15.107:0/875526632 >> v1:172.21.15.195:6800/2984408421 conn(0x7f151c06ded0 legacy=0x7f151c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 10 monclient: shutdown 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 1 -- 172.21.15.107:0/875526632 >> v1:172.21.15.195:6789/0 conn(0x7f1538061830 legacy=0x7f153813ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 1 -- 172.21.15.107:0/875526632 shutdown_connections 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 1 -- 172.21.15.107:0/875526632 >> 172.21.15.107:0/875526632 conn(0x7f15380b0ef0 msgr2=0x7f1538071190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 1 -- 172.21.15.107:0/875526632 shutdown_connections 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.015+0000 7f153fd3b640 1 -- 172.21.15.107:0/875526632 wait complete. 2024-03-25T19:46:29.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:29.016+0000 7f153fd3b640 1 librados: shutdown 2024-03-25T19:46:29.029 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:31.561 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:01.877 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:31.614 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:01.930 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:31.647 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:01.963 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:31.657 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:01.944 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:31.665 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:01.952 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:31.680 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:01.996 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:31.693 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:02.010 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:31.699 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:01.986 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:31.719 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:02.035 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:35.835 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:35.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.947+0000 7f95f1a2a640 10 monclient: get_monmap_and_config 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.947+0000 7f95f1a2a640 10 monclient: build_initial_monmap 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.948+0000 7f95f1a2a640 10 monclient: monmap: 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:46:35.949977+0000 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:46:35.949977+0000 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:46:35.949 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:35.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.948+0000 7f95f1a2a640 1 Processor -- start 2024-03-25T19:46:35.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.948+0000 7f95f1a2a640 1 -- start start 2024-03-25T19:46:35.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.948+0000 7f95f1a2a640 10 monclient: init 2024-03-25T19:46:35.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.949+0000 7f95f1a2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:35.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.949+0000 7f95f1a2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:35.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.949+0000 7f95f1a2a640 10 monclient(hunting): picked mon.noname-c con 0x7f95ec0a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:35.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.949+0000 7f95f1a2a640 10 monclient(hunting): picked mon.noname-a con 0x7f95ec13f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:35.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.949+0000 7f95f1a2a640 10 monclient(hunting): picked mon.noname-b con 0x7f95ec1430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:35.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.949+0000 7f95f1a2a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec142520 con 0x7f95ec13f710 2024-03-25T19:46:35.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.949+0000 7f95f1a2a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0b28a0 con 0x7f95ec0a98f0 2024-03-25T19:46:35.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.949+0000 7f95f1a2a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0b2a30 con 0x7f95ec1430c0 2024-03-25T19:46:35.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.949+0000 7f95f1a2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:35.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.949+0000 7f95f1a2a640 10 monclient(hunting): authenticate will time out at 5188.291360s 2024-03-25T19:46:35.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.962+0000 7f95eb7fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f95ec1430c0 0x7f95ec14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55242/0 (socket says 172.21.15.107:55242) 2024-03-25T19:46:35.962 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.962+0000 7f95eb7fe640 1 -- 172.21.15.107:0/2666240081 learned_addr learned my addr 172.21.15.107:0/2666240081 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:35.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.976+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291590192 0 0) 0x7f95ec0b2a30 con 0x7f95ec1430c0 2024-03-25T19:46:35.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.976+0000 7f95e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.976+0000 7f95e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.976+0000 7f95e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.976+0000 7f95e9ffb640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:46:35.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.977+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d4003520 con 0x7f95ec1430c0 2024-03-25T19:46:35.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.977+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951473370 0 0) 0x7f95ec142520 con 0x7f95ec13f710 2024-03-25T19:46:35.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.977+0000 7f95e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.977+0000 7f95e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.977+0000 7f95e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.977+0000 7f95e9ffb640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:46:35.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.977+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec0b2a30 con 0x7f95ec13f710 2024-03-25T19:46:35.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.981+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975237437 0 0) 0x7f95ec0b28a0 con 0x7f95ec0a98f0 2024-03-25T19:46:35.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.981+0000 7f95e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.981+0000 7f95e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.981+0000 7f95e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.981+0000 7f95e9ffb640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:46:35.982 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.981+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec142520 con 0x7f95ec0a98f0 2024-03-25T19:46:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1418645983 0 0) 0x7f95ec0b2a30 con 0x7f95ec13f710 2024-03-25T19:46:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 >> v1:172.21.15.107:6790/0 conn(0x7f95ec1430c0 legacy=0x7f95ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 >> v1:172.21.15.195:6789/0 conn(0x7f95ec0a98f0 legacy=0x7f95ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 1 monclient: found mon.noname-a 2024-03-25T19:46:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:35.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0b44c0 con 0x7f95ec13f710 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f95e0001670 con 0x7f95ec13f710 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95e9ffb640 10 monclient: dump: 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.986 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:35.987 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:35.987 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:35.987 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:35.987 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:35.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95f1a2a640 5 monclient: authenticate success, global_id 4704 2024-03-25T19:46:35.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:35.984+0000 7f95f1a2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:36.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.002+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95e00017d0 con 0x7f95ec13f710 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.002+0000 7f95e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.002+0000 7f95e9ffb640 1 -- 172.21.15.107:0/2666240081 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f95e0001be0 con 0x7f95ec13f710 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.002+0000 7f95f1a2a640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.002+0000 7f95f1a2a640 4 set_mon_vals no callback set 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.002+0000 7f95e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.002+0000 7f95e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.002+0000 7f95e9ffb640 10 monclient: dump: 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:36.003 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:36.004 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:36.004 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:36.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.003+0000 7f95f1a2a640 10 monclient: shutdown 2024-03-25T19:46:36.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.003+0000 7f95f1a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:36.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.003+0000 7f95f1a2a640 1 -- 172.21.15.107:0/2666240081 >> v1:172.21.15.107:6789/0 conn(0x7f95ec13f710 legacy=0x7f95ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:36.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.003+0000 7f95f1a2a640 1 -- 172.21.15.107:0/2666240081 shutdown_connections 2024-03-25T19:46:36.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.003+0000 7f95f1a2a640 1 -- 172.21.15.107:0/2666240081 >> 172.21.15.107:0/2666240081 conn(0x7f95ec0b7ea0 msgr2=0x7f95ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:36.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.003+0000 7f95f1a2a640 1 -- 172.21.15.107:0/2666240081 shutdown_connections 2024-03-25T19:46:36.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.003+0000 7f95f1a2a640 1 -- 172.21.15.107:0/2666240081 wait complete. 2024-03-25T19:46:36.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.003+0000 7f95f1a2a640 10 monclient: build_initial_monmap 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.003+0000 7f95f1a2a640 10 monclient: monmap: 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.004+0000 7f95f1a2a640 1 librados: starting msgr at 2024-03-25T19:46:36.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.004+0000 7f95f1a2a640 1 librados: starting objecter 2024-03-25T19:46:36.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.004+0000 7f95f1a2a640 1 Processor -- start 2024-03-25T19:46:36.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.004+0000 7f95f1a2a640 1 -- start start 2024-03-25T19:46:36.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.004+0000 7f95f1a2a640 1 librados: setting wanted keys 2024-03-25T19:46:36.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.004+0000 7f95f1a2a640 1 librados: calling monclient init 2024-03-25T19:46:36.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.004+0000 7f95f1a2a640 10 monclient: init 2024-03-25T19:46:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.005+0000 7f95f1a2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.005+0000 7f95f1a2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.005+0000 7f95f1a2a640 10 monclient(hunting): picked mon.noname-a con 0x7f95ec0a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.005+0000 7f95f1a2a640 10 monclient(hunting): picked mon.noname-c con 0x7f95ec13f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.005+0000 7f95f1a2a640 10 monclient(hunting): picked mon.noname-b con 0x7f95ec1430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.005+0000 7f95f1a2a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec1398e0 con 0x7f95ec0a98f0 2024-03-25T19:46:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.005+0000 7f95f1a2a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0ab8b0 con 0x7f95ec1430c0 2024-03-25T19:46:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.005+0000 7f95f1a2a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0abb90 con 0x7f95ec13f710 2024-03-25T19:46:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.005+0000 7f95f1a2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:36.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.005+0000 7f95f1a2a640 10 monclient(hunting): authenticate will time out at 5188.347354s 2024-03-25T19:46:36.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.014+0000 7f95eaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f95ec0a98f0 0x7f95ec134e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36684/0 (socket says 172.21.15.107:36684) 2024-03-25T19:46:36.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.014+0000 7f95eaffd640 1 -- 172.21.15.107:0/1975007555 learned_addr learned my addr 172.21.15.107:0/1975007555 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:36.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.015+0000 7f95d3fff640 10 client.?.objecter ms_handle_connect 0x7f95ec0a98f0 2024-03-25T19:46:36.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.015+0000 7f95d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:36.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.016+0000 7f95d3fff640 10 client.?.objecter ms_handle_connect 0x7f95ec13f710 2024-03-25T19:46:36.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.016+0000 7f95d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:36.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.025+0000 7f95d3fff640 10 client.?.objecter ms_handle_connect 0x7f95ec1430c0 2024-03-25T19:46:36.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.025+0000 7f95d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:36.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.032+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069970334 0 0) 0x7f95ec0abb90 con 0x7f95ec13f710 2024-03-25T19:46:36.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.032+0000 7f95d3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:36.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.032+0000 7f95d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:36.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.032+0000 7f95d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:36.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.032+0000 7f95d3fff640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:46:36.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.032+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d8004eb0 con 0x7f95ec13f710 2024-03-25T19:46:36.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.034+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822430954 0 0) 0x7f95ec1398e0 con 0x7f95ec0a98f0 2024-03-25T19:46:36.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.034+0000 7f95d3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:36.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.034+0000 7f95d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:36.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.034+0000 7f95d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:36.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.034+0000 7f95d3fff640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:46:36.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.034+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec0abb90 con 0x7f95ec0a98f0 2024-03-25T19:46:36.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.041+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192024197 0 0) 0x7f95ec0ab8b0 con 0x7f95ec1430c0 2024-03-25T19:46:36.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.041+0000 7f95d3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:36.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.041+0000 7f95d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:36.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.041+0000 7f95d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:36.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.041+0000 7f95d3fff640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:46:36.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.041+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec1398e0 con 0x7f95ec1430c0 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.056+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2222979986 0 0) 0x7f95ec1398e0 con 0x7f95ec1430c0 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.056+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95ec0ab8b0 con 0x7f95ec1430c0 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.056+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f95e4001a30 con 0x7f95ec1430c0 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.056+0000 7f95d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.056+0000 7f95d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.056+0000 7f95d3fff640 10 monclient(hunting): dump: 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:36.057 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:36.058 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:36.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.056+0000 7f95d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:36.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.058+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3538784886 0 0) 0x7f95d8004eb0 con 0x7f95ec13f710 2024-03-25T19:46:36.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.058+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95ec1398e0 con 0x7f95ec13f710 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.058+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f95e0001ba0 con 0x7f95ec13f710 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.058+0000 7f95d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.058+0000 7f95d3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.058+0000 7f95d3fff640 10 monclient(hunting): dump: 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:36.059 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:36.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1492234484 0 0) 0x7f95ec1398e0 con 0x7f95ec13f710 2024-03-25T19:46:36.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 >> v1:172.21.15.195:6789/0 conn(0x7f95ec1430c0 legacy=0x7f95ec1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:36.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 >> v1:172.21.15.107:6789/0 conn(0x7f95ec0a98f0 legacy=0x7f95ec134e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:36.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95d3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:36.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95d3fff640 1 monclient: found mon.c 2024-03-25T19:46:36.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:36.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95d3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:46:36.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec138a20 con 0x7f95ec13f710 2024-03-25T19:46:36.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95d3fff640 10 monclient: _finish_auth 0 2024-03-25T19:46:36.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95d3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:36.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:36.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95f1a2a640 5 monclient: authenticate success, global_id 4709 2024-03-25T19:46:36.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95f1a2a640 10 monclient: _renew_subs 2024-03-25T19:46:36.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95f1a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:46:36.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95f1a2a640 1 -- 172.21.15.107:0/1975007555 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec137840 con 0x7f95ec13f710 2024-03-25T19:46:36.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95f1a2a640 10 client.4709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:36.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95f1a2a640 10 monclient: _renew_subs 2024-03-25T19:46:36.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95f1a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:46:36.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95f1a2a640 1 -- 172.21.15.107:0/1975007555 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec138500 con 0x7f95ec13f710 2024-03-25T19:46:36.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.069+0000 7f95f1a2a640 1 librados: init done 2024-03-25T19:46:36.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.070+0000 7f95f1a2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:36.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.070+0000 7f95f1a2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:36.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.070+0000 7f95f1a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:46:36.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.070+0000 7f95f1a2a640 1 -- 172.21.15.107:0/1975007555 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec0b27a0 con 0x7f95ec13f710 2024-03-25T19:46:36.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.413+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95e00038a0 con 0x7f95ec13f710 2024-03-25T19:46:36.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.413+0000 7f95d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:36.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.413+0000 7f95e97fa640 4 set_mon_vals no callback set 2024-03-25T19:46:36.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.413+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f95e0003bb0 con 0x7f95ec13f710 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.413+0000 7f95d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.413+0000 7f95d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.413+0000 7f95d3fff640 10 monclient: dump: 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:36.415 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:36.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.414+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f95e0019b80 con 0x7f95ec13f710 2024-03-25T19:46:36.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.414+0000 7f95eaffd640 10 monclient: get_auth_request con 0x7f95d806dfd0 auth_method 0 2024-03-25T19:46:36.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.415+0000 7f95d3fff640 10 client.4709.objecter ms_handle_connect 0x7f95d806dfd0 2024-03-25T19:46:36.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.420+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3155620770 0 0) 0x7f95e00537c0 con 0x7f95ec13f710 2024-03-25T19:46:36.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.420+0000 7f95d3fff640 10 client.4709.objecter ms_dispatch 0x7f95ec000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:36.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.420+0000 7f95d3fff640 3 client.4709.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:36.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.420+0000 7f95d3fff640 3 client.4709.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:36.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.420+0000 7f95d3fff640 20 client.4709.objecter dump_active .. 0 homeless 2024-03-25T19:46:36.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.420+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95e004f100 con 0x7f95ec13f710 2024-03-25T19:46:36.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.420+0000 7f95d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:36.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.420+0000 7f95d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:36.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.627+0000 7f95f1a2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:36.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.627+0000 7f95f1a2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:36.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.627+0000 7f95f1a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:46:36.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.627+0000 7f95f1a2a640 1 -- 172.21.15.107:0/1975007555 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95ec138bb0 con 0x7f95ec13f710 2024-03-25T19:46:36.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.695+0000 7f95d3fff640 1 -- 172.21.15.107:0/1975007555 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2864573051) 0x7f95e0051200 con 0x7f95ec13f710 2024-03-25T19:46:36.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.695+0000 7f95d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:36.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.695+0000 7f95d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:36.697 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:46:36.697 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:45:51.343698+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:51.343698+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:36.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.698+0000 7f95f1a2a640 10 librados: watch_flush enter 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.698+0000 7f95f1a2a640 10 librados: watch_flush exit 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.698+0000 7f95f1a2a640 20 client.4709.objecter shutdown clearing up homeless session... 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.698+0000 7f95f1a2a640 10 client.4709.objecter successfully canceled tick 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.698+0000 7f95f1a2a640 1 -- 172.21.15.107:0/1975007555 >> v1:172.21.15.195:6800/2984408421 conn(0x7f95d806dfd0 legacy=0x7f95d8070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.698+0000 7f95f1a2a640 10 monclient: shutdown 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.698+0000 7f95f1a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.698+0000 7f95f1a2a640 1 -- 172.21.15.107:0/1975007555 >> v1:172.21.15.107:6790/0 conn(0x7f95ec13f710 legacy=0x7f95ec1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.698+0000 7f95f1a2a640 1 -- 172.21.15.107:0/1975007555 shutdown_connections 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.698+0000 7f95f1a2a640 1 -- 172.21.15.107:0/1975007555 >> 172.21.15.107:0/1975007555 conn(0x7f95ec0b7ea0 msgr2=0x7f95ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.698+0000 7f95f1a2a640 1 -- 172.21.15.107:0/1975007555 shutdown_connections 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.699+0000 7f95f1a2a640 1 -- 172.21.15.107:0/1975007555 wait complete. 2024-03-25T19:46:36.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:36.699+0000 7f95f1a2a640 1 librados: shutdown 2024-03-25T19:46:36.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:43.518 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:43.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.637+0000 7f16f1898640 10 monclient: get_monmap_and_config 2024-03-25T19:46:43.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.637+0000 7f16f1898640 10 monclient: build_initial_monmap 2024-03-25T19:46:43.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.637+0000 7f16f1898640 10 monclient: monmap: 2024-03-25T19:46:43.637 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:43.637 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:43.637 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:46:43.638482+0000 2024-03-25T19:46:43.638 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:46:43.638482+0000 2024-03-25T19:46:43.638 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:43.638 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:43.638 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:43.638 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:46:43.638 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:46:43.638 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:43.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.638+0000 7f16f1898640 1 Processor -- start 2024-03-25T19:46:43.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.638+0000 7f16f1898640 1 -- start start 2024-03-25T19:46:43.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.638+0000 7f16f1898640 10 monclient: init 2024-03-25T19:46:43.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.638+0000 7f16f1898640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:43.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.638+0000 7f16f1898640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:43.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.638+0000 7f16f1898640 10 monclient(hunting): picked mon.noname-b con 0x7f16ec0a98f0 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:43.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.638+0000 7f16f1898640 10 monclient(hunting): picked mon.noname-c con 0x7f16ec13f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:43.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.638+0000 7f16f1898640 10 monclient(hunting): picked mon.noname-a con 0x7f16ec1430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:43.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.638+0000 7f16f1898640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec142520 con 0x7f16ec1430c0 2024-03-25T19:46:43.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.638+0000 7f16f1898640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec0b28a0 con 0x7f16ec13f710 2024-03-25T19:46:43.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.639+0000 7f16f1898640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec0b2a30 con 0x7f16ec0a98f0 2024-03-25T19:46:43.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.639+0000 7f16f1898640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:43.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.639+0000 7f16f1898640 10 monclient(hunting): authenticate will time out at 5195.979774s 2024-03-25T19:46:43.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.661+0000 7f16e3fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f16ec13f710 0x7f16ec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:53926/0 (socket says 172.21.15.107:53926) 2024-03-25T19:46:43.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.661+0000 7f16e3fff640 1 -- 172.21.15.107:0/1239541513 learned_addr learned my addr 172.21.15.107:0/1239541513 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:43.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.677+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893853805 0 0) 0x7f16ec142520 con 0x7f16ec1430c0 2024-03-25T19:46:43.677 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.677+0000 7f16ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:43.677 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.677+0000 7f16ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:43.677 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.677+0000 7f16ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:43.677 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.677+0000 7f16ea7fc640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:46:43.677 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.677+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d8003520 con 0x7f16ec1430c0 2024-03-25T19:46:43.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.678+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557136714 0 0) 0x7f16ec0b28a0 con 0x7f16ec13f710 2024-03-25T19:46:43.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.678+0000 7f16ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:43.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.678+0000 7f16ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:43.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.678+0000 7f16ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:43.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.678+0000 7f16ea7fc640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:46:43.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.678+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16ec142520 con 0x7f16ec13f710 2024-03-25T19:46:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419655545 0 0) 0x7f16ec0b2a30 con 0x7f16ec0a98f0 2024-03-25T19:46:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:46:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16ec0b28a0 con 0x7f16ec0a98f0 2024-03-25T19:46:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 42975455 0 0) 0x7f16ec142520 con 0x7f16ec13f710 2024-03-25T19:46:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 >> v1:172.21.15.107:6790/0 conn(0x7f16ec0a98f0 legacy=0x7f16ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 >> v1:172.21.15.107:6789/0 conn(0x7f16ec1430c0 legacy=0x7f16ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 1 monclient: found mon.noname-c 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec0b44c0 con 0x7f16ec13f710 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f16d4001670 con 0x7f16ec13f710 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16ea7fc640 10 monclient: dump: 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:43.689 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:43.690 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:43.690 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:43.690 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:43.690 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:43.690 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16f1898640 5 monclient: authenticate success, global_id 4702 2024-03-25T19:46:43.690 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.688+0000 7f16f1898640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:43.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16d40017d0 con 0x7f16ec13f710 2024-03-25T19:46:43.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:43.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16ea7fc640 1 -- 172.21.15.107:0/1239541513 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f16d4001be0 con 0x7f16ec13f710 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16f1898640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16f1898640 4 set_mon_vals no callback set 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16ea7fc640 10 monclient: dump: 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:43.702 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:43.703 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:43.703 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:43.703 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:43.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16f1898640 10 monclient: shutdown 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16f1898640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16f1898640 1 -- 172.21.15.107:0/1239541513 >> v1:172.21.15.195:6789/0 conn(0x7f16ec13f710 legacy=0x7f16ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16f1898640 1 -- 172.21.15.107:0/1239541513 shutdown_connections 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.702+0000 7f16f1898640 1 -- 172.21.15.107:0/1239541513 >> 172.21.15.107:0/1239541513 conn(0x7f16ec0b7ea0 msgr2=0x7f16ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.703+0000 7f16f1898640 1 -- 172.21.15.107:0/1239541513 shutdown_connections 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.703+0000 7f16f1898640 1 -- 172.21.15.107:0/1239541513 wait complete. 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.703+0000 7f16f1898640 10 monclient: build_initial_monmap 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.703+0000 7f16f1898640 10 monclient: monmap: 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:43.705 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.704+0000 7f16f1898640 1 librados: starting msgr at 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.704+0000 7f16f1898640 1 librados: starting objecter 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.704+0000 7f16f1898640 1 Processor -- start 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.704+0000 7f16f1898640 1 -- start start 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.704+0000 7f16f1898640 1 librados: setting wanted keys 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.704+0000 7f16f1898640 1 librados: calling monclient init 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.704+0000 7f16f1898640 10 monclient: init 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.705+0000 7f16f1898640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.705+0000 7f16f1898640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.705+0000 7f16f1898640 10 monclient(hunting): picked mon.noname-b con 0x7f16ec0a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.705+0000 7f16f1898640 10 monclient(hunting): picked mon.noname-c con 0x7f16ec13f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.705+0000 7f16f1898640 10 monclient(hunting): picked mon.noname-a con 0x7f16ec1430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:43.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.705+0000 7f16f1898640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec1390f0 con 0x7f16ec1430c0 2024-03-25T19:46:43.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.705+0000 7f16f1898640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec138dd0 con 0x7f16ec0a98f0 2024-03-25T19:46:43.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.705+0000 7f16f1898640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec1346b0 con 0x7f16ec13f710 2024-03-25T19:46:43.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.705+0000 7f16f1898640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:43.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.705+0000 7f16f1898640 10 monclient(hunting): authenticate will time out at 5196.045925s 2024-03-25T19:46:43.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.709+0000 7f16eaffd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f16ec0a98f0 0x7f16ec137e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:53964/0 (socket says 172.21.15.107:53964) 2024-03-25T19:46:43.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.709+0000 7f16eaffd640 1 -- 172.21.15.107:0/3754726438 learned_addr learned my addr 172.21.15.107:0/3754726438 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.711+0000 7f16e8ff9640 10 client.?.objecter ms_handle_connect 0x7f16ec13f710 2024-03-25T19:46:43.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.711+0000 7f16e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:43.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.717+0000 7f16e8ff9640 10 client.?.objecter ms_handle_connect 0x7f16ec0a98f0 2024-03-25T19:46:43.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.717+0000 7f16e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:43.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.724+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681682082 0 0) 0x7f16ec1346b0 con 0x7f16ec13f710 2024-03-25T19:46:43.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.724+0000 7f16e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:43.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.724+0000 7f16e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:43.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.724+0000 7f16e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:43.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.724+0000 7f16e8ff9640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:46:43.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.724+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16cc004eb0 con 0x7f16ec13f710 2024-03-25T19:46:43.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.727+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581061056 0 0) 0x7f16ec138dd0 con 0x7f16ec0a98f0 2024-03-25T19:46:43.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.727+0000 7f16e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:43.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.727+0000 7f16e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:43.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.727+0000 7f16e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:43.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.727+0000 7f16e8ff9640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:46:43.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.727+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16ec1346b0 con 0x7f16ec0a98f0 2024-03-25T19:46:43.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.739+0000 7f16e8ff9640 10 client.?.objecter ms_handle_connect 0x7f16ec1430c0 2024-03-25T19:46:43.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.739+0000 7f16e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:43.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.744+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 891384150 0 0) 0x7f16cc004eb0 con 0x7f16ec13f710 2024-03-25T19:46:43.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.744+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16ec138dd0 con 0x7f16ec13f710 2024-03-25T19:46:43.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.744+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f16d4001ba0 con 0x7f16ec13f710 2024-03-25T19:46:43.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.744+0000 7f16e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:43.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.744+0000 7f16e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:43.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.744+0000 7f16e8ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:43.744 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:43.744 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:43.745 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:43.745 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:43.745 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:43.745 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:43.745 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:43.745 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:43.745 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:43.745 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:43.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.744+0000 7f16e8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:43.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2551533060 0 0) 0x7f16ec138dd0 con 0x7f16ec13f710 2024-03-25T19:46:43.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 >> v1:172.21.15.195:6789/0 conn(0x7f16ec0a98f0 legacy=0x7f16ec137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 >> v1:172.21.15.107:6789/0 conn(0x7f16ec1430c0 legacy=0x7f16ec133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16e8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:43.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16e8ff9640 1 monclient: found mon.c 2024-03-25T19:46:43.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:43.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16e8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:46:43.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec134840 con 0x7f16ec13f710 2024-03-25T19:46:43.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16e8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:43.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16e8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:43.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:43.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16f1898640 5 monclient: authenticate success, global_id 4715 2024-03-25T19:46:43.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16f1898640 10 monclient: _renew_subs 2024-03-25T19:46:43.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16f1898640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:46:43.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16f1898640 1 -- 172.21.15.107:0/3754726438 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec134a90 con 0x7f16ec13f710 2024-03-25T19:46:43.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16f1898640 10 client.4715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:43.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16f1898640 10 monclient: _renew_subs 2024-03-25T19:46:43.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16f1898640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:46:43.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16f1898640 1 -- 172.21.15.107:0/3754726438 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec1356f0 con 0x7f16ec13f710 2024-03-25T19:46:43.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.750+0000 7f16f1898640 1 librados: init done 2024-03-25T19:46:43.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.751+0000 7f16f1898640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:43.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.751+0000 7f16f1898640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:43.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.751+0000 7f16f1898640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:46:43.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:43.751+0000 7f16f1898640 1 -- 172.21.15.107:0/3754726438 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec0627b0 con 0x7f16ec13f710 2024-03-25T19:46:44.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.051+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16d4004a90 con 0x7f16ec13f710 2024-03-25T19:46:44.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.051+0000 7f16e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:44.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.051+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f16d4004da0 con 0x7f16ec13f710 2024-03-25T19:46:44.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.051+0000 7f16e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:44.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.051+0000 7f16e8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:44.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.051+0000 7f16e8ff9640 10 monclient: dump: 2024-03-25T19:46:44.052 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:44.052 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:44.052 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:44.052 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:44.052 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:44.052 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:44.052 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:44.052 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:44.052 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:44.053 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:44.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.051+0000 7f16e9ffb640 4 set_mon_vals no callback set 2024-03-25T19:46:44.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.052+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f16d401ad70 con 0x7f16ec13f710 2024-03-25T19:46:44.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.053+0000 7f16eaffd640 10 monclient: get_auth_request con 0x7f16cc06dd10 auth_method 0 2024-03-25T19:46:44.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.053+0000 7f16e8ff9640 10 client.4715.objecter ms_handle_connect 0x7f16cc06dd10 2024-03-25T19:46:44.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.060+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3155620770 0 0) 0x7f16d40548b0 con 0x7f16ec13f710 2024-03-25T19:46:44.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.060+0000 7f16e8ff9640 10 client.4715.objecter ms_dispatch 0x7f16ec000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:44.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.060+0000 7f16e8ff9640 3 client.4715.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:44.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.060+0000 7f16e8ff9640 3 client.4715.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:44.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.060+0000 7f16e8ff9640 20 client.4715.objecter dump_active .. 0 homeless 2024-03-25T19:46:44.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.060+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16d40503c0 con 0x7f16ec13f710 2024-03-25T19:46:44.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.060+0000 7f16e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:44.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.060+0000 7f16e8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:44.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.323+0000 7f16f1898640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:44.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.323+0000 7f16f1898640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:44.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.323+0000 7f16f1898640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-25T19:46:44.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.323+0000 7f16f1898640 1 -- 172.21.15.107:0/3754726438 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16ec1361b0 con 0x7f16ec13f710 2024-03-25T19:46:44.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.373+0000 7f16e8ff9640 1 -- 172.21.15.107:0/3754726438 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2864573051) 0x7f16d40522f0 con 0x7f16ec13f710 2024-03-25T19:46:44.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.373+0000 7f16e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:44.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.373+0000 7f16e8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:44.373 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:46:44.373 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:45:51.343698+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:51.343698+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:44.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 10 librados: watch_flush enter 2024-03-25T19:46:44.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 10 librados: watch_flush exit 2024-03-25T19:46:44.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 20 client.4715.objecter shutdown clearing up homeless session... 2024-03-25T19:46:44.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 10 client.4715.objecter successfully canceled tick 2024-03-25T19:46:44.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 1 -- 172.21.15.107:0/3754726438 >> v1:172.21.15.195:6800/2984408421 conn(0x7f16cc06dd10 legacy=0x7f16cc0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:44.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 10 monclient: shutdown 2024-03-25T19:46:44.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:44.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 1 -- 172.21.15.107:0/3754726438 >> v1:172.21.15.107:6790/0 conn(0x7f16ec13f710 legacy=0x7f16ec1386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:44.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 1 -- 172.21.15.107:0/3754726438 shutdown_connections 2024-03-25T19:46:44.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 1 -- 172.21.15.107:0/3754726438 >> 172.21.15.107:0/3754726438 conn(0x7f16ec0b7ea0 msgr2=0x7f16ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:44.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 1 -- 172.21.15.107:0/3754726438 shutdown_connections 2024-03-25T19:46:44.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 1 -- 172.21.15.107:0/3754726438 wait complete. 2024-03-25T19:46:44.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:44.376+0000 7f16f1898640 1 librados: shutdown 2024-03-25T19:46:44.388 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:51.195 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:51.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.300+0000 7f52e800d640 10 monclient: get_monmap_and_config 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.300+0000 7f52e800d640 10 monclient: build_initial_monmap 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.300+0000 7f52e800d640 10 monclient: monmap: 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:46:51.301125+0000 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:46:51.301125+0000 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:46:51.300 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:51.301 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.300+0000 7f52e800d640 1 Processor -- start 2024-03-25T19:46:51.301 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.301+0000 7f52e800d640 1 -- start start 2024-03-25T19:46:51.301 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.301+0000 7f52e800d640 10 monclient: init 2024-03-25T19:46:51.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.301+0000 7f52e800d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:51.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.301+0000 7f52e800d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:51.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.301+0000 7f52e800d640 10 monclient(hunting): picked mon.noname-c con 0x7f52e00a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:51.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.301+0000 7f52e800d640 10 monclient(hunting): picked mon.noname-b con 0x7f52e013f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:51.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.301+0000 7f52e800d640 10 monclient(hunting): picked mon.noname-a con 0x7f52e01430c0 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:51.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.302+0000 7f52e800d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e0142520 con 0x7f52e01430c0 2024-03-25T19:46:51.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.302+0000 7f52e800d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e00b28a0 con 0x7f52e00a98f0 2024-03-25T19:46:51.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.302+0000 7f52e800d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e00b2a30 con 0x7f52e013f710 2024-03-25T19:46:51.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.302+0000 7f52e800d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:51.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.302+0000 7f52e800d640 10 monclient(hunting): authenticate will time out at 5203.642797s 2024-03-25T19:46:51.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.309+0000 7f52e5da9640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f52e00a98f0 0x7f52e00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:48756/0 (socket says 172.21.15.107:48756) 2024-03-25T19:46:51.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.309+0000 7f52e5da9640 1 -- 172.21.15.107:0/4087668300 learned_addr learned my addr 172.21.15.107:0/4087668300 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:51.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.323+0000 7f52e4da7640 1 -- 172.21.15.107:0/4087668300 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290070947 0 0) 0x7f52e0142520 con 0x7f52e01430c0 2024-03-25T19:46:51.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.323+0000 7f52e4da7640 10 monclient(hunting): _negotiate 2024-03-25T19:46:51.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.323+0000 7f52e4da7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:51.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.323+0000 7f52e4da7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:51.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.323+0000 7f52e4da7640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:46:51.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.323+0000 7f52e4da7640 1 -- 172.21.15.107:0/4087668300 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52cc003520 con 0x7f52e01430c0 2024-03-25T19:46:51.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.327+0000 7f52e4da7640 1 -- 172.21.15.107:0/4087668300 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186377840 0 0) 0x7f52e00b28a0 con 0x7f52e00a98f0 2024-03-25T19:46:51.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.327+0000 7f52e4da7640 10 monclient(hunting): _negotiate 2024-03-25T19:46:51.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.327+0000 7f52e4da7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:51.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.327+0000 7f52e4da7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:51.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.327+0000 7f52e4da7640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:46:51.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.327+0000 7f52e4da7640 1 -- 172.21.15.107:0/4087668300 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e0142520 con 0x7f52e00a98f0 2024-03-25T19:46:51.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.336+0000 7f52e4da7640 1 -- 172.21.15.107:0/4087668300 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2678639695 0 0) 0x7f52cc003520 con 0x7f52e01430c0 2024-03-25T19:46:51.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 1 -- 172.21.15.107:0/4087668300 >> v1:172.21.15.107:6790/0 conn(0x7f52e013f710 legacy=0x7f52e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 1 -- 172.21.15.107:0/4087668300 >> v1:172.21.15.195:6789/0 conn(0x7f52e00a98f0 legacy=0x7f52e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 10 monclient: _finish_hunting 0 2024-03-25T19:46:51.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 1 monclient: found mon.noname-a 2024-03-25T19:46:51.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:51.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:46:51.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 1 -- 172.21.15.107:0/4087668300 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e00b44c0 con 0x7f52e01430c0 2024-03-25T19:46:51.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 10 monclient: _finish_auth 0 2024-03-25T19:46:51.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 10 monclient: _check_auth_tickets 2024-03-25T19:46:51.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:51.337 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 1 -- 172.21.15.107:0/4087668300 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f52d8001a10 con 0x7f52e01430c0 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e4da7640 10 monclient: dump: 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e800d640 5 monclient: authenticate success, global_id 4716 2024-03-25T19:46:51.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.337+0000 7f52e800d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:51.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e4da7640 1 -- 172.21.15.107:0/4087668300 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52d8001b70 con 0x7f52e01430c0 2024-03-25T19:46:51.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e4da7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:51.344 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e4da7640 1 -- 172.21.15.107:0/4087668300 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f52d8001f80 con 0x7f52e01430c0 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e4da7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e4da7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e4da7640 10 monclient: dump: 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e800d640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:51.345 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e800d640 4 set_mon_vals no callback set 2024-03-25T19:46:51.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e800d640 10 monclient: shutdown 2024-03-25T19:46:51.347 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e800d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e800d640 1 -- 172.21.15.107:0/4087668300 >> v1:172.21.15.107:6789/0 conn(0x7f52e01430c0 legacy=0x7f52e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e800d640 1 -- 172.21.15.107:0/4087668300 shutdown_connections 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.345+0000 7f52e800d640 1 -- 172.21.15.107:0/4087668300 >> 172.21.15.107:0/4087668300 conn(0x7f52e00b7ea0 msgr2=0x7f52e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.346+0000 7f52e800d640 1 -- 172.21.15.107:0/4087668300 shutdown_connections 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.346+0000 7f52e800d640 1 -- 172.21.15.107:0/4087668300 wait complete. 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.346+0000 7f52e800d640 10 monclient: build_initial_monmap 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.346+0000 7f52e800d640 10 monclient: monmap: 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:46:51.348 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.347+0000 7f52e800d640 1 librados: starting msgr at 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.347+0000 7f52e800d640 1 librados: starting objecter 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.347+0000 7f52e800d640 1 Processor -- start 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.347+0000 7f52e800d640 1 -- start start 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.347+0000 7f52e800d640 1 librados: setting wanted keys 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.347+0000 7f52e800d640 1 librados: calling monclient init 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.347+0000 7f52e800d640 10 monclient: init 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.347+0000 7f52e800d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.348+0000 7f52e800d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.348+0000 7f52e800d640 10 monclient(hunting): picked mon.noname-b con 0x7f52e00a98f0 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.348+0000 7f52e800d640 10 monclient(hunting): picked mon.noname-a con 0x7f52e013f710 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:51.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.348+0000 7f52e800d640 10 monclient(hunting): picked mon.noname-c con 0x7f52e01430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:51.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.348+0000 7f52e800d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e0136c50 con 0x7f52e013f710 2024-03-25T19:46:51.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.348+0000 7f52e800d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e0136ec0 con 0x7f52e00a98f0 2024-03-25T19:46:51.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.348+0000 7f52e800d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e01317f0 con 0x7f52e01430c0 2024-03-25T19:46:51.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.348+0000 7f52e800d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:51.350 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.348+0000 7f52e800d640 10 monclient(hunting): authenticate will time out at 5203.688840s 2024-03-25T19:46:51.355 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.356+0000 7f52e5da9640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f52e00a98f0 0x7f52e01310e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:48782/0 (socket says 172.21.15.107:48782) 2024-03-25T19:46:51.356 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.356+0000 7f52e5da9640 1 -- 172.21.15.107:0/2695288066 learned_addr learned my addr 172.21.15.107:0/2695288066 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.358+0000 7f52d67fc640 10 client.?.objecter ms_handle_connect 0x7f52e00a98f0 2024-03-25T19:46:51.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.358+0000 7f52d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:51.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.367+0000 7f52d67fc640 10 client.?.objecter ms_handle_connect 0x7f52e013f710 2024-03-25T19:46:51.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.367+0000 7f52d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:51.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.368+0000 7f52d67fc640 10 client.?.objecter ms_handle_connect 0x7f52e01430c0 2024-03-25T19:46:51.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.368+0000 7f52d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:51.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.375+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013443877 0 0) 0x7f52e0136ec0 con 0x7f52e00a98f0 2024-03-25T19:46:51.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.375+0000 7f52d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:51.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.375+0000 7f52d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:51.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.375+0000 7f52d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:51.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.375+0000 7f52d67fc640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:46:51.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.375+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c4004eb0 con 0x7f52e00a98f0 2024-03-25T19:46:51.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.380+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762608420 0 0) 0x7f52e0136c50 con 0x7f52e013f710 2024-03-25T19:46:51.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.380+0000 7f52d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:51.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.380+0000 7f52d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:51.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.380+0000 7f52d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:51.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.380+0000 7f52d67fc640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:46:51.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.380+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e0136ec0 con 0x7f52e013f710 2024-03-25T19:46:51.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.381+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92085936 0 0) 0x7f52e01317f0 con 0x7f52e01430c0 2024-03-25T19:46:51.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.381+0000 7f52d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:51.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.381+0000 7f52d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:51.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.381+0000 7f52d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:51.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.381+0000 7f52d67fc640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:46:51.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.381+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e0136c50 con 0x7f52e01430c0 2024-03-25T19:46:51.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.384+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 864298463 0 0) 0x7f52c4004eb0 con 0x7f52e00a98f0 2024-03-25T19:46:51.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.384+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52e01317f0 con 0x7f52e00a98f0 2024-03-25T19:46:51.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.386+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f52dc0019b0 con 0x7f52e00a98f0 2024-03-25T19:46:51.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.386+0000 7f52d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:51.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.386+0000 7f52d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:51.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.386+0000 7f52d67fc640 10 monclient(hunting): dump: 2024-03-25T19:46:51.386 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:51.386 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:51.386 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:51.386 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:51.387 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:51.387 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:51.387 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:51.387 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:51.387 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:51.387 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:51.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.386+0000 7f52d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.394+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1168016727 0 0) 0x7f52e01317f0 con 0x7f52e00a98f0 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 >> v1:172.21.15.107:6790/0 conn(0x7f52e01430c0 legacy=0x7f52e01db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 >> v1:172.21.15.107:6789/0 conn(0x7f52e013f710 legacy=0x7f52e01dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52d67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52d67fc640 1 monclient: found mon.b 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52d67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e0131980 con 0x7f52e00a98f0 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52d67fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52d67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52e800d640 5 monclient: authenticate success, global_id 4711 2024-03-25T19:46:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52e800d640 10 monclient: _renew_subs 2024-03-25T19:46:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52e800d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52e800d640 1 -- 172.21.15.107:0/2695288066 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e0135070 con 0x7f52e00a98f0 2024-03-25T19:46:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52e800d640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52e800d640 10 monclient: _renew_subs 2024-03-25T19:46:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52e800d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52e800d640 1 -- 172.21.15.107:0/2695288066 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e00b7870 con 0x7f52e00a98f0 2024-03-25T19:46:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.395+0000 7f52e800d640 1 librados: init done 2024-03-25T19:46:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.396+0000 7f52e800d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.396+0000 7f52e800d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.396+0000 7f52e800d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.396+0000 7f52e800d640 1 -- 172.21.15.107:0/2695288066 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e00627b0 con 0x7f52e00a98f0 2024-03-25T19:46:51.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.681+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52dc0047e0 con 0x7f52e00a98f0 2024-03-25T19:46:51.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.681+0000 7f52d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:51.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.681+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f52dc004b70 con 0x7f52e00a98f0 2024-03-25T19:46:51.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.681+0000 7f52d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:51.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.681+0000 7f52d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:51.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.681+0000 7f52d67fc640 10 monclient: dump: 2024-03-25T19:46:51.682 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:51.682 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:51.682 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:51.682 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:51.682 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:51.682 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:51.682 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:51.682 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:51.682 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:51.682 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:51.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.681+0000 7f52d7fff640 4 set_mon_vals no callback set 2024-03-25T19:46:51.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.682+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f52dc004d90 con 0x7f52e00a98f0 2024-03-25T19:46:51.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.683+0000 7f52e55a8640 10 monclient: get_auth_request con 0x7f52c406ddb0 auth_method 0 2024-03-25T19:46:51.683 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.683+0000 7f52d67fc640 10 client.4711.objecter ms_handle_connect 0x7f52c406ddb0 2024-03-25T19:46:51.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.687+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3155620770 0 0) 0x7f52dc0548c0 con 0x7f52e00a98f0 2024-03-25T19:46:51.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.687+0000 7f52d67fc640 10 client.4711.objecter ms_dispatch 0x7f52e0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:51.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.687+0000 7f52d67fc640 3 client.4711.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:51.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.687+0000 7f52d67fc640 3 client.4711.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:51.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.687+0000 7f52d67fc640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T19:46:51.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.688+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52dc0503d0 con 0x7f52e00a98f0 2024-03-25T19:46:51.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.688+0000 7f52d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:51.688 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.688+0000 7f52d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:51.719 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:22.035 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:51.772 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:22.088 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:51.808 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:22.124 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:51.827 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:22.114 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:51.830 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:22.147 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:51.832 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:22.119 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:51.834 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:22.150 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:51.856 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:22.172 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:51.864 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:22.151 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:51.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.895+0000 7f52e800d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.895+0000 7f52e800d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.895+0000 7f52e800d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:46:51.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.895+0000 7f52e800d640 1 -- 172.21.15.107:0/2695288066 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52e00b7ac0 con 0x7f52e00a98f0 2024-03-25T19:46:51.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.964+0000 7f52d67fc640 1 -- 172.21.15.107:0/2695288066 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2864573051) 0x7f52dc052300 con 0x7f52e00a98f0 2024-03-25T19:46:51.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.964+0000 7f52d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.964+0000 7f52d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:51.964 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:46:51.964 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:45:51.343698+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:51.343698+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:51.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 10 librados: watch_flush enter 2024-03-25T19:46:51.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 10 librados: watch_flush exit 2024-03-25T19:46:51.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T19:46:51.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 10 client.4711.objecter successfully canceled tick 2024-03-25T19:46:51.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 1 -- 172.21.15.107:0/2695288066 >> v1:172.21.15.195:6800/2984408421 conn(0x7f52c406ddb0 legacy=0x7f52c4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 10 monclient: shutdown 2024-03-25T19:46:51.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:51.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 1 -- 172.21.15.107:0/2695288066 >> v1:172.21.15.195:6789/0 conn(0x7f52e00a98f0 legacy=0x7f52e01310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 1 -- 172.21.15.107:0/2695288066 shutdown_connections 2024-03-25T19:46:51.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 1 -- 172.21.15.107:0/2695288066 >> 172.21.15.107:0/2695288066 conn(0x7f52e00b7ea0 msgr2=0x7f52e013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:51.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 1 -- 172.21.15.107:0/2695288066 shutdown_connections 2024-03-25T19:46:51.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 1 -- 172.21.15.107:0/2695288066 wait complete. 2024-03-25T19:46:51.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:51.966+0000 7f52e800d640 1 librados: shutdown 2024-03-25T19:46:51.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:56.515 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:26.831 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:56.524 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:26.840 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:56.531 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:26.847 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:56.535 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:26.851 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:56.535 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:26.822 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:56.538 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:26.824 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:56.544 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:26.861 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:56.554 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:26.870 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:56.556 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:26.843 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:58.784 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:58.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.897+0000 7f823ef1c640 10 monclient: get_monmap_and_config 2024-03-25T19:46:58.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.897+0000 7f823ef1c640 10 monclient: build_initial_monmap 2024-03-25T19:46:58.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.897+0000 7f823ef1c640 10 monclient: monmap: 2024-03-25T19:46:58.898 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:58.898 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:58.898 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:46:58.898817+0000 2024-03-25T19:46:58.898 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:46:58.898817+0000 2024-03-25T19:46:58.898 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:58.898 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:58.898 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:58.898 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:46:58.898 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:46:58.898 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.898+0000 7f823ef1c640 1 Processor -- start 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.898+0000 7f823ef1c640 1 -- start start 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.898+0000 7f823ef1c640 10 monclient: init 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.899+0000 7f823ef1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.899+0000 7f823ef1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.899+0000 7f823ef1c640 10 monclient(hunting): picked mon.noname-c con 0x7f82380b4d60 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.899+0000 7f823ef1c640 10 monclient(hunting): picked mon.noname-b con 0x7f82380b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.899+0000 7f823ef1c640 10 monclient(hunting): picked mon.noname-a con 0x7f823814be90 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.899+0000 7f823ef1c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82380af940 con 0x7f823814be90 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.899+0000 7f823ef1c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82380b0a70 con 0x7f82380b4d60 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.899+0000 7f823ef1c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82380b0d50 con 0x7f82380b1730 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.899+0000 7f823ef1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:58.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.899+0000 7f823ef1c640 10 monclient(hunting): authenticate will time out at 5211.240317s 2024-03-25T19:46:58.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.910+0000 7f8237fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f82380b1730 0x7f82380b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:33796/0 (socket says 172.21.15.107:33796) 2024-03-25T19:46:58.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.911+0000 7f8237fff640 1 -- 172.21.15.107:0/2132720425 learned_addr learned my addr 172.21.15.107:0/2132720425 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:58.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.926+0000 7f82377fe640 1 -- 172.21.15.107:0/2132720425 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221869752 0 0) 0x7f82380b0d50 con 0x7f82380b1730 2024-03-25T19:46:58.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.926+0000 7f82377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.926+0000 7f82377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.926+0000 7f82377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.926+0000 7f82377fe640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:46:58.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.926+0000 7f82377fe640 1 -- 172.21.15.107:0/2132720425 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8228003520 con 0x7f82380b1730 2024-03-25T19:46:58.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 1 -- 172.21.15.107:0/2132720425 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2605607302 0 0) 0x7f8228003520 con 0x7f82380b1730 2024-03-25T19:46:58.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 1 -- 172.21.15.107:0/2132720425 >> v1:172.21.15.195:6789/0 conn(0x7f82380b4d60 legacy=0x7f823813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 1 -- 172.21.15.107:0/2132720425 >> v1:172.21.15.107:6789/0 conn(0x7f823814be90 legacy=0x7f823814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:58.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 1 monclient: found mon.noname-b 2024-03-25T19:46:58.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:58.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-25T19:46:58.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 1 -- 172.21.15.107:0/2132720425 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82380b1030 con 0x7f82380b1730 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 1 -- 172.21.15.107:0/2132720425 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8224001690 con 0x7f82380b1730 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f82377fe640 10 monclient: dump: 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:58.940 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:58.941 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:58.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f823ef1c640 5 monclient: authenticate success, global_id 4724 2024-03-25T19:46:58.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.939+0000 7f823ef1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:58.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.954+0000 7f82377fe640 1 -- 172.21.15.107:0/2132720425 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82240017f0 con 0x7f82380b1730 2024-03-25T19:46:58.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.954+0000 7f82377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:58.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.954+0000 7f82377fe640 1 -- 172.21.15.107:0/2132720425 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8224001c00 con 0x7f82380b1730 2024-03-25T19:46:58.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.954+0000 7f823ef1c640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:58.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.954+0000 7f823ef1c640 4 set_mon_vals no callback set 2024-03-25T19:46:58.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.954+0000 7f82377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.954+0000 7f82377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:58.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.954+0000 7f82377fe640 10 monclient: dump: 2024-03-25T19:46:58.955 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:58.955 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:58.955 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:58.955 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:58.955 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.955 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:58.955 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:58.955 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:58.955 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:58.955 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:58.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.955+0000 7f823ef1c640 10 monclient: shutdown 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.955+0000 7f823ef1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.955+0000 7f823ef1c640 1 -- 172.21.15.107:0/2132720425 >> v1:172.21.15.107:6790/0 conn(0x7f82380b1730 legacy=0x7f82380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.955+0000 7f823ef1c640 1 -- 172.21.15.107:0/2132720425 shutdown_connections 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.955+0000 7f823ef1c640 1 -- 172.21.15.107:0/2132720425 >> 172.21.15.107:0/2132720425 conn(0x7f82380617a0 msgr2=0x7f8238061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.955+0000 7f823ef1c640 1 -- 172.21.15.107:0/2132720425 shutdown_connections 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.955+0000 7f823ef1c640 1 -- 172.21.15.107:0/2132720425 wait complete. 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.956+0000 7f823ef1c640 10 monclient: build_initial_monmap 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.956+0000 7f823ef1c640 10 monclient: monmap: 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:46:58.957 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:46:58.958 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:46:58.958 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:58.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.956+0000 7f823ef1c640 1 librados: starting msgr at 2024-03-25T19:46:58.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.956+0000 7f823ef1c640 1 librados: starting objecter 2024-03-25T19:46:58.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.956+0000 7f823ef1c640 1 Processor -- start 2024-03-25T19:46:58.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.956+0000 7f823ef1c640 1 -- start start 2024-03-25T19:46:58.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.956+0000 7f823ef1c640 1 librados: setting wanted keys 2024-03-25T19:46:58.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.956+0000 7f823ef1c640 1 librados: calling monclient init 2024-03-25T19:46:58.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.956+0000 7f823ef1c640 10 monclient: init 2024-03-25T19:46:58.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.957+0000 7f823ef1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:58.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.957+0000 7f823ef1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:58.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.957+0000 7f823ef1c640 10 monclient(hunting): picked mon.noname-b con 0x7f82380b1730 addr v1:172.21.15.195:6789/0 2024-03-25T19:46:58.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.957+0000 7f823ef1c640 10 monclient(hunting): picked mon.noname-a con 0x7f82380b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:46:58.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.957+0000 7f823ef1c640 10 monclient(hunting): picked mon.noname-c con 0x7f823814be90 addr v1:172.21.15.107:6790/0 2024-03-25T19:46:58.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.957+0000 7f823ef1c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8238136250 con 0x7f82380b4d60 2024-03-25T19:46:58.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.957+0000 7f823ef1c640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8238136af0 con 0x7f82380b1730 2024-03-25T19:46:58.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.957+0000 7f823ef1c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8238136c80 con 0x7f823814be90 2024-03-25T19:46:58.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.957+0000 7f823ef1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:58.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.957+0000 7f823ef1c640 10 monclient(hunting): authenticate will time out at 5211.298717s 2024-03-25T19:46:58.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.965+0000 7f823ccb8640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f82380b1730 0x7f82381d7620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:52964/0 (socket says 172.21.15.107:52964) 2024-03-25T19:46:58.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.965+0000 7f823ccb8640 1 -- 172.21.15.107:0/3041856003 learned_addr learned my addr 172.21.15.107:0/3041856003 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:46:58.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.967+0000 7f82357fa640 10 client.?.objecter ms_handle_connect 0x7f82380b1730 2024-03-25T19:46:58.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.967+0000 7f82357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:58.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.969+0000 7f82357fa640 10 client.?.objecter ms_handle_connect 0x7f823814be90 2024-03-25T19:46:58.969 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.969+0000 7f82357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:58.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.976+0000 7f82357fa640 10 client.?.objecter ms_handle_connect 0x7f82380b4d60 2024-03-25T19:46:58.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.976+0000 7f82357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:58.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.980+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542267438 0 0) 0x7f8238136c80 con 0x7f823814be90 2024-03-25T19:46:58.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.980+0000 7f82357fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.980+0000 7f82357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.980+0000 7f82357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.980+0000 7f82357fa640 10 monclient(hunting): my global_id is 4727 2024-03-25T19:46:58.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.980+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f821c004eb0 con 0x7f823814be90 2024-03-25T19:46:58.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.987+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832615548 0 0) 0x7f8238136250 con 0x7f82380b4d60 2024-03-25T19:46:58.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.987+0000 7f82357fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.987+0000 7f82357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.987+0000 7f82357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.987+0000 7f82357fa640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:46:58.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.988+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8238136c80 con 0x7f82380b4d60 2024-03-25T19:46:58.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.991+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304213211 0 0) 0x7f8238136af0 con 0x7f82380b1730 2024-03-25T19:46:58.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.991+0000 7f82357fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.991+0000 7f82357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.991+0000 7f82357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.991+0000 7f82357fa640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:46:58.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.991+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8238136250 con 0x7f82380b1730 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.995+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2484184803 0 0) 0x7f821c004eb0 con 0x7f823814be90 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.995+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8238136af0 con 0x7f823814be90 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.995+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8230001b90 con 0x7f823814be90 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.995+0000 7f82357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.995+0000 7f82357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.995+0000 7f82357fa640 10 monclient(hunting): dump: 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.995 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:58.996 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:58.996 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:58.996 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:58.996 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:58.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:58.995+0000 7f82357fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:59.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.003+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3516410963 0 0) 0x7f8238136c80 con 0x7f82380b4d60 2024-03-25T19:46:59.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.003+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f821c004eb0 con 0x7f82380b4d60 2024-03-25T19:46:59.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.003+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8224001b40 con 0x7f82380b4d60 2024-03-25T19:46:59.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.003+0000 7f82357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:59.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.003+0000 7f82357fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:59.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.003+0000 7f82357fa640 10 monclient(hunting): dump: 2024-03-25T19:46:59.003 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:59.003 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:59.003 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:59.003 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:59.003 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:59.004 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:59.004 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:59.004 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:59.004 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:59.004 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:59.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 678389594 0 0) 0x7f821c004eb0 con 0x7f82380b4d60 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 >> v1:172.21.15.107:6790/0 conn(0x7f823814be90 legacy=0x7f82381e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 >> v1:172.21.15.195:6789/0 conn(0x7f82380b1730 legacy=0x7f82381d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f82357fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f82357fa640 1 monclient: found mon.a 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f82357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f82357fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8238138210 con 0x7f82380b4d60 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f82357fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f82357fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f82357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f823ef1c640 5 monclient: authenticate success, global_id 4725 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f823ef1c640 10 monclient: _renew_subs 2024-03-25T19:46:59.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f823ef1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:46:59.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f823ef1c640 1 -- 172.21.15.107:0/3041856003 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82381384c0 con 0x7f82380b4d60 2024-03-25T19:46:59.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f823ef1c640 10 client.4725.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:59.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f823ef1c640 10 monclient: _renew_subs 2024-03-25T19:46:59.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f823ef1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:46:59.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f823ef1c640 1 -- 172.21.15.107:0/3041856003 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8238145700 con 0x7f82380b4d60 2024-03-25T19:46:59.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.008+0000 7f823ef1c640 1 librados: init done 2024-03-25T19:46:59.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.009+0000 7f823ef1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:59.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.009+0000 7f823ef1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:59.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.009+0000 7f823ef1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:46:59.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.009+0000 7f823ef1c640 1 -- 172.21.15.107:0/3041856003 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8238135f70 con 0x7f82380b4d60 2024-03-25T19:46:59.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.316+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8224004a90 con 0x7f82380b4d60 2024-03-25T19:46:59.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.316+0000 7f82357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:59.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.316+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f8224001250 con 0x7f82380b4d60 2024-03-25T19:46:59.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.316+0000 7f82357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:59.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.316+0000 7f82357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:59.316 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.316+0000 7f82357fa640 10 monclient: dump: 2024-03-25T19:46:59.316 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:46:59.316 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:46:59.316 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:59.316 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:46:59.317 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:59.317 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:46:59.317 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:46:59.317 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:46:59.317 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:46:59.317 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:46:59.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.316+0000 7f8236ffd640 4 set_mon_vals no callback set 2024-03-25T19:46:59.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.317+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f822401aa70 con 0x7f82380b4d60 2024-03-25T19:46:59.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.317+0000 7f823ccb8640 10 monclient: get_auth_request con 0x7f821c06dfb0 auth_method 0 2024-03-25T19:46:59.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.317+0000 7f82357fa640 10 client.4725.objecter ms_handle_connect 0x7f821c06dfb0 2024-03-25T19:46:59.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.323+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3155620770 0 0) 0x7f82240546a0 con 0x7f82380b4d60 2024-03-25T19:46:59.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.323+0000 7f82357fa640 10 client.4725.objecter ms_dispatch 0x7f8238000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:59.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.323+0000 7f82357fa640 3 client.4725.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:59.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.323+0000 7f82357fa640 3 client.4725.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:59.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.323+0000 7f82357fa640 20 client.4725.objecter dump_active .. 0 homeless 2024-03-25T19:46:59.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.323+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f822404ffe0 con 0x7f82380b4d60 2024-03-25T19:46:59.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.323+0000 7f82357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:59.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.323+0000 7f82357fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:59.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.574+0000 7f823ef1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:59.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.574+0000 7f823ef1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:59.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.574+0000 7f823ef1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:46:59.575 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.574+0000 7f823ef1c640 1 -- 172.21.15.107:0/3041856003 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82380627b0 con 0x7f82380b4d60 2024-03-25T19:46:59.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.612+0000 7f82357fa640 1 -- 172.21.15.107:0/3041856003 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2864573051) 0x7f82240520e0 con 0x7f82380b4d60 2024-03-25T19:46:59.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.612+0000 7f82357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:59.613 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.612+0000 7f82357fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:59.613 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:46:59.613 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:45:51.343698+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:51.343698+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:59.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 10 librados: watch_flush enter 2024-03-25T19:46:59.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 10 librados: watch_flush exit 2024-03-25T19:46:59.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 20 client.4725.objecter shutdown clearing up homeless session... 2024-03-25T19:46:59.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 10 client.4725.objecter successfully canceled tick 2024-03-25T19:46:59.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 1 -- 172.21.15.107:0/3041856003 >> v1:172.21.15.195:6800/2984408421 conn(0x7f821c06dfb0 legacy=0x7f821c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:59.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 10 monclient: shutdown 2024-03-25T19:46:59.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:59.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 1 -- 172.21.15.107:0/3041856003 >> v1:172.21.15.107:6789/0 conn(0x7f82380b4d60 legacy=0x7f823812fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:59.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 1 -- 172.21.15.107:0/3041856003 shutdown_connections 2024-03-25T19:46:59.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 1 -- 172.21.15.107:0/3041856003 >> 172.21.15.107:0/3041856003 conn(0x7f82380617a0 msgr2=0x7f82380b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:59.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 1 -- 172.21.15.107:0/3041856003 shutdown_connections 2024-03-25T19:46:59.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 1 -- 172.21.15.107:0/3041856003 wait complete. 2024-03-25T19:46:59.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:46:59.615+0000 7f823ef1c640 1 librados: shutdown 2024-03-25T19:46:59.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:06.434 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:06.550 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.550+0000 7f2858493640 10 monclient: get_monmap_and_config 2024-03-25T19:47:06.550 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.550+0000 7f2858493640 10 monclient: build_initial_monmap 2024-03-25T19:47:06.550 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.550+0000 7f2858493640 10 monclient: monmap: 2024-03-25T19:47:06.550 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:47:06.550 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:06.551 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:47:06.551774+0000 2024-03-25T19:47:06.551 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:47:06.551774+0000 2024-03-25T19:47:06.551 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:06.551 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:06.551 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:47:06.551 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:47:06.551 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:47:06.551 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:06.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.551+0000 7f2858493640 1 Processor -- start 2024-03-25T19:47:06.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.551+0000 7f2858493640 1 -- start start 2024-03-25T19:47:06.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.551+0000 7f2858493640 10 monclient: init 2024-03-25T19:47:06.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.552+0000 7f2858493640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:06.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.552+0000 7f2858493640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:06.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.552+0000 7f2858493640 10 monclient(hunting): picked mon.noname-b con 0x7f28500b4d60 addr v1:172.21.15.107:6790/0 2024-03-25T19:47:06.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.552+0000 7f2858493640 10 monclient(hunting): picked mon.noname-a con 0x7f28500b1730 addr v1:172.21.15.107:6789/0 2024-03-25T19:47:06.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.552+0000 7f2858493640 10 monclient(hunting): picked mon.noname-c con 0x7f285014be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:47:06.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.552+0000 7f2858493640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28500af940 con 0x7f28500b1730 2024-03-25T19:47:06.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.552+0000 7f2858493640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28500b0a70 con 0x7f285014be90 2024-03-25T19:47:06.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.552+0000 7f2858493640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28500b0d50 con 0x7f28500b4d60 2024-03-25T19:47:06.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.552+0000 7f2858493640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:06.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.552+0000 7f2858493640 10 monclient(hunting): authenticate will time out at 5218.893149s 2024-03-25T19:47:06.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.557+0000 7f285622f640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f28500b4d60 0x7f285013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36522/0 (socket says 172.21.15.107:36522) 2024-03-25T19:47:06.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.557+0000 7f285622f640 1 -- 172.21.15.107:0/3260420446 learned_addr learned my addr 172.21.15.107:0/3260420446 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:47:06.575 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.576+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772489856 0 0) 0x7f28500af940 con 0x7f28500b1730 2024-03-25T19:47:06.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.576+0000 7f285522d640 10 monclient(hunting): _negotiate 2024-03-25T19:47:06.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.576+0000 7f285522d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:06.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.576+0000 7f285522d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:06.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.576+0000 7f285522d640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:47:06.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.576+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c003520 con 0x7f28500b1730 2024-03-25T19:47:06.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.577+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331491939 0 0) 0x7f28500b0d50 con 0x7f28500b4d60 2024-03-25T19:47:06.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.577+0000 7f285522d640 10 monclient(hunting): _negotiate 2024-03-25T19:47:06.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.577+0000 7f285522d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:06.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.577+0000 7f285522d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:06.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.577+0000 7f285522d640 10 monclient(hunting): my global_id is 4730 2024-03-25T19:47:06.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.577+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28500af940 con 0x7f28500b4d60 2024-03-25T19:47:06.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.587+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879188008 0 0) 0x7f28500b0a70 con 0x7f285014be90 2024-03-25T19:47:06.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.587+0000 7f285522d640 10 monclient(hunting): _negotiate 2024-03-25T19:47:06.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.587+0000 7f285522d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:06.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.587+0000 7f285522d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:06.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.587+0000 7f285522d640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:47:06.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.587+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28500b0d50 con 0x7f285014be90 2024-03-25T19:47:06.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.590+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754041068 0 0) 0x7f283c003520 con 0x7f28500b1730 2024-03-25T19:47:06.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 >> v1:172.21.15.107:6790/0 conn(0x7f28500b4d60 legacy=0x7f285013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 >> v1:172.21.15.195:6789/0 conn(0x7f285014be90 legacy=0x7f285014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 10 monclient: _finish_hunting 0 2024-03-25T19:47:06.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 1 monclient: found mon.noname-a 2024-03-25T19:47:06.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:06.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-25T19:47:06.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28500b1030 con 0x7f28500b1730 2024-03-25T19:47:06.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 10 monclient: _finish_auth 0 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 10 monclient: _check_auth_tickets 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2840001690 con 0x7f28500b1730 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f2858493640 5 monclient: authenticate success, global_id 4728 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f2858493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f285522d640 10 monclient: dump: 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.592 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:06.593 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:47:06.593 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:47:06.593 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:47:06.593 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:06.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.591+0000 7f2858493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:06.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.597+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28400017f0 con 0x7f28500b1730 2024-03-25T19:47:06.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.597+0000 7f285522d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:06.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.597+0000 7f285522d640 1 -- 172.21.15.107:0/3260420446 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2840001c00 con 0x7f28500b1730 2024-03-25T19:47:06.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.597+0000 7f285522d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.597+0000 7f285522d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:06.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.597+0000 7f285522d640 10 monclient: dump: 2024-03-25T19:47:06.597 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:47:06.597 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:06.597 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:06.597 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:06.597 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.598 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:06.598 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:47:06.598 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:47:06.598 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:47:06.598 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:06.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.597+0000 7f2858493640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:06.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.597+0000 7f2858493640 4 set_mon_vals no callback set 2024-03-25T19:47:06.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.598+0000 7f2858493640 10 monclient: shutdown 2024-03-25T19:47:06.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.598+0000 7f2858493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.598+0000 7f2858493640 1 -- 172.21.15.107:0/3260420446 >> v1:172.21.15.107:6789/0 conn(0x7f28500b1730 legacy=0x7f28500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.598+0000 7f2858493640 1 -- 172.21.15.107:0/3260420446 shutdown_connections 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.598+0000 7f2858493640 1 -- 172.21.15.107:0/3260420446 >> 172.21.15.107:0/3260420446 conn(0x7f28500617a0 msgr2=0x7f2850061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.598+0000 7f2858493640 1 -- 172.21.15.107:0/3260420446 shutdown_connections 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.598+0000 7f2858493640 1 -- 172.21.15.107:0/3260420446 wait complete. 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.599+0000 7f2858493640 10 monclient: build_initial_monmap 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.599+0000 7f2858493640 10 monclient: monmap: 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:06.600 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:47:06.601 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:47:06.601 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:47:06.601 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:47:06.601 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:06.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.599+0000 7f2858493640 1 librados: starting msgr at 2024-03-25T19:47:06.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.599+0000 7f2858493640 1 librados: starting objecter 2024-03-25T19:47:06.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 1 Processor -- start 2024-03-25T19:47:06.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 1 -- start start 2024-03-25T19:47:06.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 1 librados: setting wanted keys 2024-03-25T19:47:06.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 1 librados: calling monclient init 2024-03-25T19:47:06.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 10 monclient: init 2024-03-25T19:47:06.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:06.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:06.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 10 monclient(hunting): picked mon.noname-c con 0x7f28500b1730 addr v1:172.21.15.107:6790/0 2024-03-25T19:47:06.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 10 monclient(hunting): picked mon.noname-a con 0x7f28500b4d60 addr v1:172.21.15.107:6789/0 2024-03-25T19:47:06.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 10 monclient(hunting): picked mon.noname-b con 0x7f285014be90 addr v1:172.21.15.195:6789/0 2024-03-25T19:47:06.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2850141f10 con 0x7f28500b4d60 2024-03-25T19:47:06.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2850140da0 con 0x7f285014be90 2024-03-25T19:47:06.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2850138860 con 0x7f28500b1730 2024-03-25T19:47:06.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:06.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.600+0000 7f2858493640 10 monclient(hunting): authenticate will time out at 5218.941587s 2024-03-25T19:47:06.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.609+0000 7f2856a30640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f285014be90 0x7f28501e2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.107:46546/0 (socket says 172.21.15.107:46546) 2024-03-25T19:47:06.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.609+0000 7f2856a30640 1 -- 172.21.15.107:0/696403111 learned_addr learned my addr 172.21.15.107:0/696403111 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:47:06.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.611+0000 7f285622f640 1 -- 172.21.15.107:0/696403111 >> v1:172.21.15.107:6790/0 conn(0x7f28500b1730 legacy=0x7f28501405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:47:06.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.611+0000 7f285622f640 1 -- 172.21.15.107:0/696403111 >> v1:172.21.15.107:6790/0 conn(0x7f28500b1730 legacy=0x7f28501405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:47:06.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.611+0000 7f285622f640 1 --1- 172.21.15.107:0/696403111 >> v1:172.21.15.107:6790/0 conn(0x7f28500b1730 0x7f28501405f0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T19:47:06.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.611+0000 7f285622f640 1 --1- 172.21.15.107:0/696403111 >> v1:172.21.15.107:6790/0 conn(0x7f28500b1730 0x7f28501405f0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T19:47:06.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.611+0000 7f2846ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.107:6790/0 2024-03-25T19:47:06.613 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.613+0000 7f2846ffd640 10 client.?.objecter ms_handle_connect 0x7f28500b4d60 2024-03-25T19:47:06.613 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.613+0000 7f2846ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:06.613 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.614+0000 7f2846ffd640 10 client.?.objecter ms_handle_connect 0x7f285014be90 2024-03-25T19:47:06.613 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.614+0000 7f2846ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:06.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.629+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200392904 0 0) 0x7f2850141f10 con 0x7f28500b4d60 2024-03-25T19:47:06.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.629+0000 7f2846ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:06.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.629+0000 7f2846ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:06.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.629+0000 7f2846ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:06.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.629+0000 7f2846ffd640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:47:06.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.629+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2834004eb0 con 0x7f28500b4d60 2024-03-25T19:47:06.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.633+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899971527 0 0) 0x7f2850140da0 con 0x7f285014be90 2024-03-25T19:47:06.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.633+0000 7f2846ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:06.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.633+0000 7f2846ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:06.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.633+0000 7f2846ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:06.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.633+0000 7f2846ffd640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:47:06.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.633+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2850141f10 con 0x7f285014be90 2024-03-25T19:47:06.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.645+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4116506175 0 0) 0x7f2834004eb0 con 0x7f28500b4d60 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.645+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2850140da0 con 0x7f28500b4d60 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.645+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2840001b00 con 0x7f28500b4d60 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.645+0000 7f2846ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.645+0000 7f2846ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.645+0000 7f2846ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:47:06.645 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:47:06.646 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:06.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.645+0000 7f2846ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:06.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.650+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3019563594 0 0) 0x7f2850141f10 con 0x7f285014be90 2024-03-25T19:47:06.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.650+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2834004eb0 con 0x7f285014be90 2024-03-25T19:47:06.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.650+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2848001c50 con 0x7f285014be90 2024-03-25T19:47:06.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.650+0000 7f2846ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.650+0000 7f2846ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:06.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.650+0000 7f2846ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:06.651 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:47:06.651 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:06.651 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:06.651 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:06.651 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.651 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:06.651 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:47:06.651 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:47:06.651 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:47:06.651 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:06.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3577519711 0 0) 0x7f2850140da0 con 0x7f28500b4d60 2024-03-25T19:47:06.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 >> v1:172.21.15.107:6790/0 conn(0x7f28500b1730 legacy=0x7f28501405f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:47:06.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 >> v1:172.21.15.195:6789/0 conn(0x7f285014be90 legacy=0x7f28501e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2846ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:06.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2846ffd640 1 monclient: found mon.a 2024-03-25T19:47:06.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2846ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:06.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2846ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:47:06.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850137f50 con 0x7f28500b4d60 2024-03-25T19:47:06.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2846ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:06.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2846ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:06.665 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2846ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:06.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2858493640 5 monclient: authenticate success, global_id 4731 2024-03-25T19:47:06.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2858493640 10 monclient: _renew_subs 2024-03-25T19:47:06.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2858493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:47:06.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2858493640 1 -- 172.21.15.107:0/696403111 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28501d8080 con 0x7f28500b4d60 2024-03-25T19:47:06.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2858493640 10 client.4731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:06.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2858493640 10 monclient: _renew_subs 2024-03-25T19:47:06.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2858493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:47:06.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2858493640 1 -- 172.21.15.107:0/696403111 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28501d85b0 con 0x7f28500b4d60 2024-03-25T19:47:06.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.665+0000 7f2858493640 1 librados: init done 2024-03-25T19:47:06.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.666+0000 7f2858493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:06.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.666+0000 7f2858493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:06.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.666+0000 7f2858493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:47:06.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.666+0000 7f2858493640 1 -- 172.21.15.107:0/696403111 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28501381a0 con 0x7f28500b4d60 2024-03-25T19:47:06.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.886+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2840001220 con 0x7f28500b4d60 2024-03-25T19:47:06.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.886+0000 7f2846ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:06.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.886+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f2840003a40 con 0x7f28500b4d60 2024-03-25T19:47:06.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.886+0000 7f2846ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.886+0000 7f2846ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:06.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.886+0000 7f2846ffd640 10 monclient: dump: 2024-03-25T19:47:06.886 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:47:06.886 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:06.886 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:06.886 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:06.886 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.886 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:06.887 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:47:06.887 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:47:06.887 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:47:06.887 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:06.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.886+0000 7f2854a2c640 4 set_mon_vals no callback set 2024-03-25T19:47:06.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.959+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f2840019a10 con 0x7f28500b4d60 2024-03-25T19:47:06.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.960+0000 7f285622f640 10 monclient: get_auth_request con 0x7f283406df20 auth_method 0 2024-03-25T19:47:06.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.960+0000 7f2846ffd640 10 client.4731.objecter ms_handle_connect 0x7f283406df20 2024-03-25T19:47:06.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.967+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3155620770 0 0) 0x7f2840053690 con 0x7f28500b4d60 2024-03-25T19:47:06.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.967+0000 7f2846ffd640 10 client.4731.objecter ms_dispatch 0x7f2850000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:47:06.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.967+0000 7f2846ffd640 3 client.4731.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:47:06.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.967+0000 7f2846ffd640 3 client.4731.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:47:06.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.967+0000 7f2846ffd640 20 client.4731.objecter dump_active .. 0 homeless 2024-03-25T19:47:06.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.967+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f284004efd0 con 0x7f28500b4d60 2024-03-25T19:47:06.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.967+0000 7f2846ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:06.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:06.967+0000 7f2846ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:07.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.211+0000 7f2858493640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:07.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.211+0000 7f2858493640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:07.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.211+0000 7f2858493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-25T19:47:07.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.211+0000 7f2858493640 1 -- 172.21.15.107:0/696403111 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28500627b0 con 0x7f28500b4d60 2024-03-25T19:47:07.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.249+0000 7f2846ffd640 1 -- 172.21.15.107:0/696403111 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2864573051) 0x7f28400510d0 con 0x7f28500b4d60 2024-03-25T19:47:07.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.249+0000 7f2846ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:07.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.249+0000 7f2846ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:07.249 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:47:07.249 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:45:51.343698+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:51.343698+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:07.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.251+0000 7f2858493640 10 librados: watch_flush enter 2024-03-25T19:47:07.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.251+0000 7f2858493640 10 librados: watch_flush exit 2024-03-25T19:47:07.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.251+0000 7f2858493640 20 client.4731.objecter shutdown clearing up homeless session... 2024-03-25T19:47:07.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.251+0000 7f2858493640 10 client.4731.objecter successfully canceled tick 2024-03-25T19:47:07.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.251+0000 7f2858493640 1 -- 172.21.15.107:0/696403111 >> v1:172.21.15.195:6800/2984408421 conn(0x7f283406df20 legacy=0x7f28340703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:07.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.251+0000 7f2858493640 10 monclient: shutdown 2024-03-25T19:47:07.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.251+0000 7f2858493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:07.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.251+0000 7f2858493640 1 -- 172.21.15.107:0/696403111 >> v1:172.21.15.107:6789/0 conn(0x7f28500b4d60 legacy=0x7f28501d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:07.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.252+0000 7f2858493640 1 -- 172.21.15.107:0/696403111 shutdown_connections 2024-03-25T19:47:07.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.252+0000 7f2858493640 1 -- 172.21.15.107:0/696403111 >> 172.21.15.107:0/696403111 conn(0x7f28500617a0 msgr2=0x7f28500b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:07.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.252+0000 7f2858493640 1 -- 172.21.15.107:0/696403111 shutdown_connections 2024-03-25T19:47:07.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.252+0000 7f2858493640 1 -- 172.21.15.107:0/696403111 wait complete. 2024-03-25T19:47:07.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:07.252+0000 7f2858493640 1 librados: shutdown 2024-03-25T19:47:07.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:11.805 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:42.120 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:11.812 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:42.128 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:11.844 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:11:42.160 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:11.873 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:42.189 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:11.879 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:42.195 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:11.880 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:42.167 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:11.883 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:42.170 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:11.909 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:11:42.225 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:11.917 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:11:42.204 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:14.070 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:14.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.171+0000 7f114427b640 10 monclient: get_monmap_and_config 2024-03-25T19:47:14.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.171+0000 7f114427b640 10 monclient: build_initial_monmap 2024-03-25T19:47:14.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.171+0000 7f114427b640 10 monclient: monmap: 2024-03-25T19:47:14.171 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:47:14.171 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:14.172 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:47:14.172642+0000 2024-03-25T19:47:14.172 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:47:14.172642+0000 2024-03-25T19:47:14.172 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:14.172 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:14.172 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:47:14.172 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-25T19:47:14.172 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-25T19:47:14.172 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:14.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.172+0000 7f114427b640 1 Processor -- start 2024-03-25T19:47:14.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.172+0000 7f114427b640 1 -- start start 2024-03-25T19:47:14.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.172+0000 7f114427b640 10 monclient: init 2024-03-25T19:47:14.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.173+0000 7f114427b640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.173+0000 7f114427b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.173+0000 7f114427b640 10 monclient(hunting): picked mon.noname-a con 0x7f113c0a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:47:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.173+0000 7f114427b640 10 monclient(hunting): picked mon.noname-c con 0x7f113c13f710 addr v1:172.21.15.195:6789/0 2024-03-25T19:47:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.173+0000 7f114427b640 10 monclient(hunting): picked mon.noname-b con 0x7f113c1430c0 addr v1:172.21.15.107:6790/0 2024-03-25T19:47:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.173+0000 7f114427b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c142520 con 0x7f113c0a98f0 2024-03-25T19:47:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.173+0000 7f114427b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0b28a0 con 0x7f113c13f710 2024-03-25T19:47:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.173+0000 7f114427b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0b2a30 con 0x7f113c1430c0 2024-03-25T19:47:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.173+0000 7f114427b640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:14.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.173+0000 7f114427b640 10 monclient(hunting): authenticate will time out at 5226.514148s 2024-03-25T19:47:14.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.185+0000 7f1142818640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f113c1430c0 0x7f113c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48160/0 (socket says 172.21.15.107:48160) 2024-03-25T19:47:14.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.185+0000 7f1142818640 1 -- 172.21.15.107:0/688289100 learned_addr learned my addr 172.21.15.107:0/688289100 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:47:14.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.205+0000 7f1141015640 1 -- 172.21.15.107:0/688289100 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51754879 0 0) 0x7f113c0b2a30 con 0x7f113c1430c0 2024-03-25T19:47:14.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.205+0000 7f1141015640 10 monclient(hunting): _negotiate 2024-03-25T19:47:14.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.205+0000 7f1141015640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:14.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.205+0000 7f1141015640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:14.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.205+0000 7f1141015640 10 monclient(hunting): my global_id is 4733 2024-03-25T19:47:14.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.205+0000 7f1141015640 1 -- 172.21.15.107:0/688289100 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1128003520 con 0x7f113c1430c0 2024-03-25T19:47:14.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.206+0000 7f1141015640 1 -- 172.21.15.107:0/688289100 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447206137 0 0) 0x7f113c0b28a0 con 0x7f113c13f710 2024-03-25T19:47:14.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.206+0000 7f1141015640 10 monclient(hunting): _negotiate 2024-03-25T19:47:14.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.206+0000 7f1141015640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:14.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.206+0000 7f1141015640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:14.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.206+0000 7f1141015640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:47:14.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.206+0000 7f1141015640 1 -- 172.21.15.107:0/688289100 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c0b2a30 con 0x7f113c13f710 2024-03-25T19:47:14.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 1 -- 172.21.15.107:0/688289100 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2066725957 0 0) 0x7f113c0b2a30 con 0x7f113c13f710 2024-03-25T19:47:14.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 1 -- 172.21.15.107:0/688289100 >> v1:172.21.15.107:6790/0 conn(0x7f113c1430c0 legacy=0x7f113c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 1 -- 172.21.15.107:0/688289100 >> v1:172.21.15.107:6789/0 conn(0x7f113c0a98f0 legacy=0x7f113c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 10 monclient: _finish_hunting 0 2024-03-25T19:47:14.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 1 monclient: found mon.noname-c 2024-03-25T19:47:14.211 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 1 -- 172.21.15.107:0/688289100 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c0b44c0 con 0x7f113c13f710 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 10 monclient: _finish_auth 0 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 10 monclient: _check_auth_tickets 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 1 -- 172.21.15.107:0/688289100 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f112c001670 con 0x7f113c13f710 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f1141015640 10 monclient: dump: 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.212 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:14.213 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:14.213 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:47:14.213 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:47:14.213 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:47:14.213 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:14.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f114427b640 5 monclient: authenticate success, global_id 4726 2024-03-25T19:47:14.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.211+0000 7f114427b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.218+0000 7f1141015640 1 -- 172.21.15.107:0/688289100 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f112c0017d0 con 0x7f113c13f710 2024-03-25T19:47:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.218+0000 7f1141015640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.218+0000 7f1141015640 1 -- 172.21.15.107:0/688289100 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f112c001ba0 con 0x7f113c13f710 2024-03-25T19:47:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.218+0000 7f1141015640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.218+0000 7f1141015640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:14.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.218+0000 7f1141015640 10 monclient: dump: 2024-03-25T19:47:14.218 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:47:14.218 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:14.219 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.219 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.219 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:14.219 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:14.219 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:47:14.219 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:47:14.219 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:47:14.219 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:14.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.218+0000 7f114427b640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:14.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.218+0000 7f114427b640 4 set_mon_vals no callback set 2024-03-25T19:47:14.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.219+0000 7f114427b640 10 monclient: shutdown 2024-03-25T19:47:14.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.219+0000 7f114427b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:14.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.219+0000 7f114427b640 1 -- 172.21.15.107:0/688289100 >> v1:172.21.15.195:6789/0 conn(0x7f113c13f710 legacy=0x7f113c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.219+0000 7f114427b640 1 -- 172.21.15.107:0/688289100 shutdown_connections 2024-03-25T19:47:14.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.219+0000 7f114427b640 1 -- 172.21.15.107:0/688289100 >> 172.21.15.107:0/688289100 conn(0x7f113c0b7ea0 msgr2=0x7f113c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:14.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.219+0000 7f114427b640 1 -- 172.21.15.107:0/688289100 shutdown_connections 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.219+0000 7f114427b640 1 -- 172.21.15.107:0/688289100 wait complete. 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.220+0000 7f114427b640 10 monclient: build_initial_monmap 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.220+0000 7f114427b640 10 monclient: monmap: 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:14.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.220+0000 7f114427b640 1 librados: starting msgr at 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.220+0000 7f114427b640 1 librados: starting objecter 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.220+0000 7f114427b640 1 Processor -- start 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.220+0000 7f114427b640 1 -- start start 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.220+0000 7f114427b640 1 librados: setting wanted keys 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.220+0000 7f114427b640 1 librados: calling monclient init 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.220+0000 7f114427b640 10 monclient: init 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.221+0000 7f114427b640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.221+0000 7f114427b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.221+0000 7f114427b640 10 monclient(hunting): picked mon.noname-a con 0x7f113c0a98f0 addr v1:172.21.15.107:6789/0 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.221+0000 7f114427b640 10 monclient(hunting): picked mon.noname-c con 0x7f113c13f710 addr v1:172.21.15.107:6790/0 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.221+0000 7f114427b640 10 monclient(hunting): picked mon.noname-b con 0x7f113c1430c0 addr v1:172.21.15.195:6789/0 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.221+0000 7f114427b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c1d78e0 con 0x7f113c0a98f0 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.221+0000 7f114427b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c1d7500 con 0x7f113c1430c0 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.221+0000 7f114427b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c1d7690 con 0x7f113c13f710 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.221+0000 7f114427b640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:14.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.221+0000 7f114427b640 10 monclient(hunting): authenticate will time out at 5226.562496s 2024-03-25T19:47:14.228 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.228+0000 7f1141816640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f113c13f710 0x7f113c1d7e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48200/0 (socket says 172.21.15.107:48200) 2024-03-25T19:47:14.228 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.228+0000 7f1141816640 1 -- 172.21.15.107:0/1412225629 learned_addr learned my addr 172.21.15.107:0/1412225629 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-25T19:47:14.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.231+0000 7f1132ffd640 10 client.?.objecter ms_handle_connect 0x7f113c1430c0 2024-03-25T19:47:14.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.231+0000 7f1132ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:14.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.231+0000 7f1132ffd640 10 client.?.objecter ms_handle_connect 0x7f113c13f710 2024-03-25T19:47:14.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.231+0000 7f1132ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:14.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.239+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652529825 0 0) 0x7f113c1d7500 con 0x7f113c1430c0 2024-03-25T19:47:14.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.239+0000 7f1132ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:14.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.239+0000 7f1132ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:14.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.239+0000 7f1132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:14.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.239+0000 7f1132ffd640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:47:14.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.239+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1120004eb0 con 0x7f113c1430c0 2024-03-25T19:47:14.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.240+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949413033 0 0) 0x7f113c1d7690 con 0x7f113c13f710 2024-03-25T19:47:14.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.240+0000 7f1132ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:14.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.240+0000 7f1132ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:14.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.240+0000 7f1132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:14.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.240+0000 7f1132ffd640 10 monclient(hunting): my global_id is 4736 2024-03-25T19:47:14.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.240+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c1d7500 con 0x7f113c13f710 2024-03-25T19:47:14.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.245+0000 7f1132ffd640 10 client.?.objecter ms_handle_connect 0x7f113c0a98f0 2024-03-25T19:47:14.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.245+0000 7f1132ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:14.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.259+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290040266 0 0) 0x7f113c1d78e0 con 0x7f113c0a98f0 2024-03-25T19:47:14.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.259+0000 7f1132ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:14.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.259+0000 7f1132ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:14.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.259+0000 7f1132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:14.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.259+0000 7f1132ffd640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:47:14.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.259+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c1d7690 con 0x7f113c0a98f0 2024-03-25T19:47:14.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.266+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3037875047 0 0) 0x7f1120004eb0 con 0x7f113c1430c0 2024-03-25T19:47:14.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.266+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f113c1d78e0 con 0x7f113c1430c0 2024-03-25T19:47:14.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.266+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1138001b90 con 0x7f113c1430c0 2024-03-25T19:47:14.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.266+0000 7f1132ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:14.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.266+0000 7f1132ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.266+0000 7f1132ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:14.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.266+0000 7f1132ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:14.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.267+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2337092683 0 0) 0x7f113c1d7500 con 0x7f113c13f710 2024-03-25T19:47:14.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.267+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1120004eb0 con 0x7f113c13f710 2024-03-25T19:47:14.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.267+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f112c001be0 con 0x7f113c13f710 2024-03-25T19:47:14.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.267+0000 7f1132ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:14.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.267+0000 7f1132ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.267+0000 7f1132ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.268+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 434029291 0 0) 0x7f113c1d7690 con 0x7f113c0a98f0 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.268+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f113c1d7500 con 0x7f113c0a98f0 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.268+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f11340017a0 con 0x7f113c0a98f0 2024-03-25T19:47:14.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.268+0000 7f1132ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.268+0000 7f1132ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.268+0000 7f1132ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:47:14.270 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:14.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1608862504 0 0) 0x7f113c1d78e0 con 0x7f113c1430c0 2024-03-25T19:47:14.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 >> v1:172.21.15.107:6790/0 conn(0x7f113c13f710 legacy=0x7f113c1d7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 >> v1:172.21.15.107:6789/0 conn(0x7f113c0a98f0 legacy=0x7f113c134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f1132ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:14.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f1132ffd640 1 monclient: found mon.b 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f1132ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f1132ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c1377c0 con 0x7f113c1430c0 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f1132ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f1132ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f1132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f114427b640 5 monclient: authenticate success, global_id 4729 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f114427b640 10 monclient: _renew_subs 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f114427b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f114427b640 1 -- 172.21.15.107:0/1412225629 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c137a70 con 0x7f113c1430c0 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f114427b640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f114427b640 10 monclient: _renew_subs 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f114427b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:47:14.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f114427b640 1 -- 172.21.15.107:0/1412225629 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c1341f0 con 0x7f113c1430c0 2024-03-25T19:47:14.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.276+0000 7f114427b640 1 librados: init done 2024-03-25T19:47:14.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.277+0000 7f114427b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:14.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.277+0000 7f114427b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:14.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.277+0000 7f114427b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:47:14.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.277+0000 7f114427b640 1 -- 172.21.15.107:0/1412225629 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113c074cc0 con 0x7f113c1430c0 2024-03-25T19:47:14.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.494+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1138001fe0 con 0x7f113c1430c0 2024-03-25T19:47:14.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.494+0000 7f1132ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:14.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.494+0000 7f1140814640 4 set_mon_vals no callback set 2024-03-25T19:47:14.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.554+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1542600268 0 0) 0x7f1138004d00 con 0x7f113c1430c0 2024-03-25T19:47:14.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.554+0000 7f1132ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:14.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.554+0000 7f1132ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:14.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.554+0000 7f1132ffd640 10 monclient: dump: 2024-03-25T19:47:14.555 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-25T19:47:14.555 INFO:teuthology.misc.health.smithi107.stderr:fsid d993d317-0537-43ba-81bc-c1cea4d4fd58 2024-03-25T19:47:14.555 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.555 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-25T19:34:58.804366+0000 2024-03-25T19:47:14.555 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:14.555 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 3 2024-03-25T19:47:14.555 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-25T19:47:14.555 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-25T19:47:14.555 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-25T19:47:14.555 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-25T19:47:14.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.555+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3615451035 0 0) 0x7f1138004e60 con 0x7f113c1430c0 2024-03-25T19:47:14.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.556+0000 7f1142017640 10 monclient: get_auth_request con 0x7f112006e0f0 auth_method 0 2024-03-25T19:47:14.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.556+0000 7f1132ffd640 10 client.4729.objecter ms_handle_connect 0x7f112006e0f0 2024-03-25T19:47:14.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.558+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3155620770 0 0) 0x7f1138054ad0 con 0x7f113c1430c0 2024-03-25T19:47:14.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.558+0000 7f1132ffd640 10 client.4729.objecter ms_dispatch 0x7f113c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:47:14.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.558+0000 7f1132ffd640 3 client.4729.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:47:14.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.558+0000 7f1132ffd640 3 client.4729.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:47:14.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.558+0000 7f1132ffd640 20 client.4729.objecter dump_active .. 0 homeless 2024-03-25T19:47:14.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.560+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1138050410 con 0x7f113c1430c0 2024-03-25T19:47:14.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.560+0000 7f1132ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:14.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.560+0000 7f1132ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:14.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.803+0000 7f114427b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.803+0000 7f114427b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.803+0000 7f114427b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-25T19:47:14.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.803+0000 7f114427b640 1 -- 172.21.15.107:0/1412225629 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f113c0abbe0 con 0x7f113c1430c0 2024-03-25T19:47:14.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.844+0000 7f1132ffd640 1 -- 172.21.15.107:0/1412225629 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2864573051) 0x7f1138052510 con 0x7f113c1430c0 2024-03-25T19:47:14.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.844+0000 7f1132ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.844+0000 7f1132ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:14.844 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-25T19:47:14.844 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"d993d317-0537-43ba-81bc-c1cea4d4fd58","created":"2024-03-25T19:35:45.876967+0000","modified":"2024-03-25T19:45:51.343698+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:51.343698+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6cd887d-dffb-40d5-8a1a-a8f4a6d472b3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"96605efd-5d37-41af-aae0-6d5ad1736b26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6d185d22-f8a7-4be4-b82c-19da029c2520","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3cfd7107-fe4d-4984-a5c3-5356cc0bfbf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c16e4fd4-2ec1-44fe-af43-ab0d1c5e0c72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d4d6f31-2da7-4634-9c90-671a6f90572e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"abea285a-e8c1-45fc-a93e-f9e979553b41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bfc712bb-31d4-489b-bd81-49e6acbac4e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:14.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.846+0000 7f114427b640 10 librados: watch_flush enter 2024-03-25T19:47:14.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 10 librados: watch_flush exit 2024-03-25T19:47:14.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 20 client.4729.objecter shutdown clearing up homeless session... 2024-03-25T19:47:14.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 10 client.4729.objecter successfully canceled tick 2024-03-25T19:47:14.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 1 -- 172.21.15.107:0/1412225629 >> v1:172.21.15.195:6800/2984408421 conn(0x7f112006e0f0 legacy=0x7f11200705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 10 monclient: shutdown 2024-03-25T19:47:14.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:14.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 1 -- 172.21.15.107:0/1412225629 >> v1:172.21.15.195:6789/0 conn(0x7f113c1430c0 legacy=0x7f113c138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 1 -- 172.21.15.107:0/1412225629 shutdown_connections 2024-03-25T19:47:14.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 1 -- 172.21.15.107:0/1412225629 >> 172.21.15.107:0/1412225629 conn(0x7f113c0b7ea0 msgr2=0x7f113c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:14.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 1 -- 172.21.15.107:0/1412225629 shutdown_connections 2024-03-25T19:47:14.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 1 -- 172.21.15.107:0/1412225629 wait complete. 2024-03-25T19:47:14.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-25T19:47:14.847+0000 7f114427b640 1 librados: shutdown 2024-03-25T19:47:14.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:14.860 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:47:14.862 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:47:14.862 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:47:14.862 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:16.762 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:11:24.970 35655== 2024-03-25T19:47:16.762 INFO:tasks.ceph.osd.0.smithi107.stderr:==00:00:11:24.970 35655== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:16.798 INFO:tasks.ceph.osd.0.smithi107.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:20.965 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:20.965 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:47:20.966 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:47:20.966 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:47:20.966 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:22.979 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:11:31.194 35654== 2024-03-25T19:47:22.979 INFO:tasks.ceph.osd.1.smithi107.stderr:==00:00:11:31.194 35654== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:23.019 INFO:tasks.ceph.osd.1.smithi107.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:27.069 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:47:27.069 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:47:27.069 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:47:27.069 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:29.041 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:11:37.245 35656== 2024-03-25T19:47:29.041 INFO:tasks.ceph.osd.2.smithi107.stderr:==00:00:11:37.245 35656== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:29.080 INFO:tasks.ceph.osd.2.smithi107.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:31.862 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:12:02.178 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:31.871 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:12:02.187 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:31.906 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:12:02.222 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:31.913 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:12:02.229 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:31.920 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:12:02.207 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:31.922 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:12:02.238 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:31.924 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:12:02.211 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:31.952 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:12:02.239 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:31.968 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:12:02.284 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:33.171 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:33.171 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:47:33.172 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:47:33.172 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:47:33.172 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:34.935 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:11:43.153 35653== 2024-03-25T19:47:34.936 INFO:tasks.ceph.osd.3.smithi107.stderr:==00:00:11:43.153 35653== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:34.976 INFO:tasks.ceph.osd.3.smithi107.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:39.274 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:47:39.274 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:47:39.275 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:47:39.275 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:41.362 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:11:49.585 36061== 2024-03-25T19:47:41.362 INFO:tasks.ceph.osd.4.smithi195.stderr:==00:00:11:49.585 36061== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:41.403 INFO:tasks.ceph.osd.4.smithi195.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:45.376 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:47:45.377 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:47:45.377 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:47:45.377 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:47.178 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:11:55.401 36062== 2024-03-25T19:47:47.179 INFO:tasks.ceph.osd.5.smithi195.stderr:==00:00:11:55.401 36062== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:47.218 INFO:tasks.ceph.osd.5.smithi195.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:51.478 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:47:51.479 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:47:51.479 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:47:51.479 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:51.905 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:12:22.221 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:51.907 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:12:22.223 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:51.924 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:12:22.240 35342== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:51.950 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:12:22.237 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:51.953 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:12:22.240 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:51.953 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:12:22.270 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:51.955 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:12:22.272 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:51.972 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:12:22.258 35518== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:51.972 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:12:22.288 35334== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:53.308 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:12:01.528 36063== 2024-03-25T19:47:53.308 INFO:tasks.ceph.osd.6.smithi195.stderr:==00:00:12:01.528 36063== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:53.361 INFO:tasks.ceph.osd.6.smithi195.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:57.581 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:57.581 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:47:57.581 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:47:57.582 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:47:57.582 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:59.000 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:12:07.217 36064== 2024-03-25T19:47:59.000 INFO:tasks.ceph.osd.7.smithi195.stderr:==00:00:12:07.217 36064== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:59.030 INFO:tasks.ceph.osd.7.smithi195.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:48:03.684 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:48:03.684 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:48:03.684 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:48:03.684 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:48:03.685 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:48:03.685 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:48:03.712 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:48:03.713 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:48:03.713 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:48:03.713 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:48:03.714+0000 7f18f256e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 35516) UID: 0 2024-03-25T19:48:03.714 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-25T19:48:03.714+0000 7f18f256e640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T19:48:03.741 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:48:03.741 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:48:03.741 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:48:03.741 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:48:03.768 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-03-25T19:48:03.758+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35307) UID: 0 2024-03-25T19:48:03.769 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-03-25T19:48:03.759+0000 9f86640 -1 mon.a@0(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:48:04.321 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:12:34.637 35334== 2024-03-25T19:48:04.321 INFO:tasks.ceph.mon.a.smithi107.stderr:==00:00:12:34.637 35334== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:48:04.330 INFO:tasks.ceph.mon.a.smithi107.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:48:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:48:09.843 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:48:09.843 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:48:09.844 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:48:09.844 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:48:09.859 INFO:tasks.ceph.mon.c.smithi107.stderr:2024-03-25T19:48:09.849+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35309) UID: 0 2024-03-25T19:48:09.860 INFO:tasks.ceph.mon.c.smithi107.stderr:2024-03-25T19:48:09.850+0000 9f86640 -1 mon.c@2(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:48:10.275 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:12:40.591 35342== 2024-03-25T19:48:10.275 INFO:tasks.ceph.mon.c.smithi107.stderr:==00:00:12:40.591 35342== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:48:10.285 INFO:tasks.ceph.mon.c.smithi107.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:48:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:48:15.945 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:48:15.946 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:48:15.946 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:48:15.946 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:48:15.962 INFO:tasks.ceph.mon.b.smithi195.stderr:2024-03-25T19:48:15.951+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35514) UID: 0 2024-03-25T19:48:15.963 INFO:tasks.ceph.mon.b.smithi195.stderr:2024-03-25T19:48:15.953+0000 9f86640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:48:16.543 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:12:46.829 35518== 2024-03-25T19:48:16.543 INFO:tasks.ceph.mon.b.smithi195.stderr:==00:00:12:46.829 35518== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:48:16.554 INFO:tasks.ceph.mon.b.smithi195.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:48:22.048 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:48:22.048 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:48:22.048 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:48:22.049 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:48:22.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:48:22.094 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-25T19:48:22.094 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:48:22.151 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-25T19:48:22.151 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:48:22.210 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-25T19:48:22.210 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:48:22.260 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-25T19:48:22.260 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:48:22.309 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-25T19:48:22.310 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:48:22.432 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-25T19:48:22.433 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:48:22.494 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-25T19:48:22.495 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:48:22.542 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-25T19:48:22.543 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:48:22.594 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:48:22.595 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331/data/mon.a.tgz 2024-03-25T19:48:22.596 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-03-25T19:48:22.611 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.wsEgthwOnO 2024-03-25T19:48:22.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.wsEgthwOnO 2024-03-25T19:48:22.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.wsEgthwOnO 2024-03-25T19:48:22.798 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.wsEgthwOnO is 132KB 2024-03-25T19:48:22.851 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.wsEgthwOnO 2024-03-25T19:48:22.865 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331/data/mon.c.tgz 2024-03-25T19:48:22.865 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-03-25T19:48:22.919 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.DEJAilzpQy 2024-03-25T19:48:22.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.DEJAilzpQy 2024-03-25T19:48:23.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.DEJAilzpQy 2024-03-25T19:48:23.091 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.DEJAilzpQy is 132KB 2024-03-25T19:48:23.143 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.DEJAilzpQy 2024-03-25T19:48:23.157 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331/data/mon.b.tgz 2024-03-25T19:48:23.157 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-03-25T19:48:23.174 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.t5gOtvhq8o 2024-03-25T19:48:23.174 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.t5gOtvhq8o 2024-03-25T19:48:23.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.t5gOtvhq8o 2024-03-25T19:48:23.379 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.t5gOtvhq8o is 132KB 2024-03-25T19:48:23.430 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.t5gOtvhq8o 2024-03-25T19:48:23.445 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:48:23.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:48:23.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:48:23.774 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:48:24.021 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:48:24.023 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331/remote/smithi107/crash 2024-03-25T19:48:24.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:48:24.052 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331/remote/smithi195/crash 2024-03-25T19:48:24.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:48:24.079 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:48:24.079 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:48:24.097 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:48:24.169 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:48:24.170 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35962.log 2024-03-25T19:48:24.170 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:48:24.170 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.35962.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:48:24.170 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:48:24.170 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35962.log.gz 2024-03-25T19:48:24.170 INFO:teuthology.orchestra.run.smithi195.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:48:24.170 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:48:24.170 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:48:24.170 INFO:teuthology.orchestra.run.smithi195.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:48:24.170 INFO:teuthology.orchestra.run.smithi195.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:48:24.170 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35831.log 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35711.log 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.35831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35922.log 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.35711.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35831.log.gz 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.35922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35791.log 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35711.log.gz 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35922.log.gz 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.35791.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:48:24.171 INFO:teuthology.orchestra.run.smithi195.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35791.log.gz 2024-03-25T19:48:24.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35671.log 2024-03-25T19:48:24.172 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:48:24.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.35671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35871.log 2024-03-25T19:48:24.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mgr.x.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.35671.log.gz 2024-03-25T19:48:24.172 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35751.log 2024-03-25T19:48:24.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.35871.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:48:24.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.35751.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:48:24.172 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35871.log.gz 2024-03-25T19:48:24.172 INFO:teuthology.orchestra.run.smithi195.stderr: 87.6%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.35751.log.gz 2024-03-25T19:48:24.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:48:24.177 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40571.log 2024-03-25T19:48:24.177 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42368.log 2024-03-25T19:48:24.177 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39634.log 2024-03-25T19:48:24.177 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40571.log: /var/log/ceph/ceph-client.admin.42368.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:48:24.177 INFO:teuthology.orchestra.run.smithi107.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40571.log.gz 2024-03-25T19:48:24.177 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41158.log 2024-03-25T19:48:24.177 INFO:teuthology.orchestra.run.smithi107.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42368.log.gz 2024-03-25T19:48:24.177 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41861.log 2024-03-25T19:48:24.177 INFO:teuthology.orchestra.run.smithi107.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39634.log.gz 2024-03-25T19:48:24.177 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41392.log 2024-03-25T19:48:24.177 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39673.log 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41158.log.gz 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40375.log 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41861.log.gz 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42056.log 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41392.log.gz 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40375.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39673.log.gz 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40727.log 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.42056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39790.log 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40375.log.gz 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42446.log 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42056.log.gz 2024-03-25T19:48:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39790.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40727.log.gz 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41314.log 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.42446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39305.log 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39790.log.gz 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40688.log 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42446.log.gz 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40219.log 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41314.log.gz 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39305.log.gz 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40688.log: /var/log/ceph/ceph-client.admin.40219.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40688.log.gz 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40219.log.gz 2024-03-25T19:48:24.179 INFO:teuthology.orchestra.run.smithi107.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40766.log 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40962.log 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41939.log 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41549.log 2024-03-25T19:48:24.180 INFO:teuthology.orchestra.run.smithi107.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40766.log.gz 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36699.log 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40962.log.gz 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35652.log 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41939.log.gz 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36699.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41549.log.gz 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42095.log 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35652.log: gzip 87.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.36699.log.gz --verbose 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr: -- /var/log/ceph/ceph-client.admin.35823.log 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.42095.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.35652.log.gz 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39985.log 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39829.log 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42095.log.gz 2024-03-25T19:48:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40454.log 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35823.log.gz 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41119.log 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39985.log.gz 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40454.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39829.log.gz -5 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39438.log 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40493.log 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40454.log.gz 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39438.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41119.log.gz 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40493.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.39438.log.gz 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41002.log 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42017.log 2024-03-25T19:48:24.182 INFO:teuthology.orchestra.run.smithi107.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40493.log.gz 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41471.log 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.42017.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41002.log.gz 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41080.log 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41666.log 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42017.log.gz 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39907.log 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41471.log.gz 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41080.log.gz 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42407.log 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41588.log 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41666.log.gz 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.42407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42485.log 2024-03-25T19:48:24.183 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39907.log.gz 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41588.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42407.log.gz 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.42485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42134.log 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41588.log.gz 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41900.log 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42485.log.gz 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.42134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41236.log 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39595.log 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42134.log.gz 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41900.log.gz 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40102.log 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39186.log 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39595.log: /var/log/ceph/ceph-client.admin.40102.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41236.log.gz 2024-03-25T19:48:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40414.log 2024-03-25T19:48:24.185 INFO:teuthology.orchestra.run.smithi107.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz 2024-03-25T19:48:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39868.log 2024-03-25T19:48:24.185 INFO:teuthology.orchestra.run.smithi107.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40102.log.gz 2024-03-25T19:48:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38976.log 2024-03-25T19:48:24.185 INFO:teuthology.orchestra.run.smithi107.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39186.log.gz 2024-03-25T19:48:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:gzip/var/log/ceph/ceph-client.admin.39868.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39946.log 2024-03-25T19:48:24.185 INFO:teuthology.orchestra.run.smithi107.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40414.log.gz 2024-03-25T19:48:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41353.log 2024-03-25T19:48:24.185 INFO:teuthology.orchestra.run.smithi107.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39868.log.gz 2024-03-25T19:48:24.185 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41275.log 2024-03-25T19:48:24.185 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38976.log.gz 2024-03-25T19:48:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41353.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39946.log.gz -5 2024-03-25T19:48:24.186 INFO:teuthology.orchestra.run.smithi107.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40923.log 2024-03-25T19:48:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40297.log 2024-03-25T19:48:24.186 INFO:teuthology.orchestra.run.smithi107.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41353.log.gz 2024-03-25T19:48:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40923.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:48:24.186 INFO:teuthology.orchestra.run.smithi107.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41275.log.gz 2024-03-25T19:48:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40297.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40923.log.gz 2024-03-25T19:48:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41978.log 2024-03-25T19:48:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.3.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40297.log.gz 2024-03-25T19:48:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39397.log 2024-03-25T19:48:24.186 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41431.log 2024-03-25T19:48:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39751.log 2024-03-25T19:48:24.187 INFO:teuthology.orchestra.run.smithi107.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41978.log.gz 2024-03-25T19:48:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39344.log 2024-03-25T19:48:24.187 INFO:teuthology.orchestra.run.smithi107.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39397.log.gz 2024-03-25T19:48:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42212.log 2024-03-25T19:48:24.187 INFO:teuthology.orchestra.run.smithi107.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41431.log.gz 2024-03-25T19:48:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39712.log 2024-03-25T19:48:24.187 INFO:teuthology.orchestra.run.smithi107.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39751.log.gz 2024-03-25T19:48:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.42212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41510.log 2024-03-25T19:48:24.187 INFO:teuthology.orchestra.run.smithi107.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39344.log.gz 2024-03-25T19:48:24.187 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42290.log 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42212.log.gz 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40180.log 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39712.log.gz 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.42290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42329.log 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41510.log.gz 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40649.log 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42290.log.gz 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:gzip/var/log/ceph/ceph-client.admin.42329.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39516.log 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40180.log.gz 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42173.log 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42329.log.gz 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41822.log 2024-03-25T19:48:24.188 INFO:teuthology.orchestra.run.smithi107.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40649.log.gz 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.42173.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39516.log.gz 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41783.log 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42173.log.gz 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-mgr.y.log: gzip 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41822.log.gz 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37718.log 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40845.log 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39477.log 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41783.log.gz 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40845.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40610.log 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37718.log.gz 2024-03-25T19:48:24.189 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40141.log 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40845.log.gz 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40805.log 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr: 87.2%/var/log/ceph/ceph-client.admin.40610.log: -- replaced with /var/log/ceph/ceph-client.admin.39477.log.gz 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40141.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40805.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29498.log 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40141.log.gz 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40610.log.gz 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.0.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40805.log.gz 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40884.log 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.tmp-client.admin.29498.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph-client.admin.35349.log 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.29498.log.gz 2024-03-25T19:48:24.190 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39266.log 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35349.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40258.log 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40884.log.gz 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39266.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35349.log.gz 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39266.log.gz 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39225.log 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40258.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39225.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39556.log 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41744.log 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39225.log.gz 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39556.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39556.log.gz 2024-03-25T19:48:24.191 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40063.log 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41744.log: 92.9%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr: 86.9%/var/log/ceph/ceph-client.admin.40063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41197.log 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41744.log.gz 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.2.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40063.log.gz 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41627.log 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42251.log 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41627.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41197.log.gz 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41627.log.gz 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40336.log 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40024.log 2024-03-25T19:48:24.192 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.42251.log: /var/log/ceph/ceph-client.admin.40336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40532.log 2024-03-25T19:48:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40024.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40336.log.gz 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42251.log.gz 2024-03-25T19:48:24.193 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:48:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41705.log 2024-03-25T19:48:24.193 INFO:teuthology.orchestra.run.smithi107.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40024.log.gz 2024-03-25T19:48:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35679.log 2024-03-25T19:48:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40532.log: /var/log/ceph/ceph-client.admin.41705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41041.log 2024-03-25T19:48:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35679.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40532.log.gz 2024-03-25T19:48:24.193 INFO:teuthology.orchestra.run.smithi107.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41705.log.gz 2024-03-25T19:48:24.193 INFO:teuthology.orchestra.run.smithi107.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.35679.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:48:24.193 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:48:24.193 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41041.log: /var/log/ceph/ceph-mon.c.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41041.log.gz 2024-03-25T19:48:24.235 INFO:teuthology.orchestra.run.smithi195.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:48:24.965 INFO:teuthology.orchestra.run.smithi107.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:48:25.054 INFO:teuthology.orchestra.run.smithi195.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:48:25.064 INFO:teuthology.orchestra.run.smithi195.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:48:25.072 INFO:teuthology.orchestra.run.smithi195.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:48:25.262 INFO:teuthology.orchestra.run.smithi107.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:48:25.268 INFO:teuthology.orchestra.run.smithi107.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:48:25.379 INFO:teuthology.orchestra.run.smithi107.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:48:25.409 INFO:teuthology.orchestra.run.smithi195.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:48:28.943 INFO:teuthology.orchestra.run.smithi107.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:48:29.466 INFO:teuthology.orchestra.run.smithi195.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m5.327s 2024-03-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi195.stderr:user 0m9.052s 2024-03-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m0.330s 2024-03-25T19:48:29.702 INFO:teuthology.orchestra.run.smithi107.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:48:29.703 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-25T19:48:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m5.595s 2024-03-25T19:48:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m14.239s 2024-03-25T19:48:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m0.549s 2024-03-25T19:48:29.704 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:48:29.704 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331/remote/smithi107/log 2024-03-25T19:48:29.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:48:32.164 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331/remote/smithi195/log 2024-03-25T19:48:32.165 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:48:33.738 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:48:33.946 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=985edf2b50264708b0e96dd52bc1e546 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:48:33.950 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:48:33.959 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:48:34.523 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:48:34.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:48:34.538 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:48:34.622 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:48:34.633 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:48:34.633 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:48:34.636 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:48:34.650 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:48:34.653 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:48:34.656 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:48:34.656 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-03-25T19:48:34.656 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 -464us[ -464us] +/- 74ms 2024-03-25T19:48:34.656 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -290us[ -298us] +/- 61ms 2024-03-25T19:48:34.656 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +726us[ +726us] +/- 64ms 2024-03-25T19:48:34.657 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:48:34.658 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:48:34.658 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-03-25T19:48:34.659 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 -482us[ -482us] +/- 74ms 2024-03-25T19:48:34.659 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -310us[ -327us] +/- 61ms 2024-03-25T19:48:34.659 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +716us[ +716us] +/- 64ms 2024-03-25T19:48:34.659 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:48:34.659 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:48:34.670 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:48:34.671 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:48:34.706 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:48:34.778 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:48:34.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:48:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:48:34.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:48:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:48:34.915 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:48:34.926 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:48:34.976 INFO:teuthology.task.internal:Duration was 4958.792445 seconds 2024-03-25T19:48:34.976 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:48:35.014 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:48:35.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:48:35.017 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:48:35.061 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:48:35.062 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:48:35.388 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:48:35.388 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-03-25T19:48:35.388 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:48:35.409 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-03-25T19:48:35.410 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:48:35.431 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:48:35.431 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:48:35.453 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:48:35.500 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:48:35.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:48:35.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:48:35.643 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:48:35.656 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:48:35.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:48:35.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:48:35.711 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:48:35.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:48:35.727 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:48:35.750 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-03-25T19:48:35.776 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-03-25T19:48:35.803 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:48:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:48:35.828 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:48:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:48:35.858 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:48:35.869 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:48:35.870 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331/remote/smithi107 2024-03-25T19:48:35.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:48:35.907 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331/remote/smithi195 2024-03-25T19:48:35.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:48:35.943 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:48:35.943 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:48:35.946 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:48:35.993 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:48:36.005 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:48:36.005 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:48:36.031 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:48:36.031 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:48:36.033 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:48:36.045 INFO:teuthology.orchestra.run.smithi107.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:48 /home/ubuntu/cephtest 2024-03-25T19:48:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:48 /home/ubuntu/cephtest 2024-03-25T19:48:36.049 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:48:36.100 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:48:36.182 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:48:36.209 DEBUG:teuthology.nuke:shortname: smithi107 2024-03-25T19:48:36.210 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:48:36.234 DEBUG:teuthology.nuke:shortname: smithi195 2024-03-25T19:48:36.235 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:48:36.238 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:13.883157', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBLuZd9QyE9I0h1TLT3GEMuC6haZ0Y7lzY5OJwwzJRByM7qKFM0qvQe2PpjZotyn3JxWXAqMDyg7SYU3euO/Z1k='} 2024-03-25T19:48:36.256 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621331', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:13.882000', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNImLuourlybiOWkqsOsHAugVHoQs9Bh5WVVf1tuCtJ0N1zw/lTL5XFajoWPCm4o+1QJQL1y04HdVYF9xa+skvU='} 2024-03-25T19:48:36.288 INFO:teuthology.orchestra.console.smithi107:Power off 2024-03-25T19:48:36.289 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:48:36.304 INFO:teuthology.orchestra.console.smithi195:Power off 2024-03-25T19:48:36.304 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:48:36.313 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-03-25T19:48:36.314 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:48:36.327 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2024-03-25T19:48:36.327 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:48:36.336 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-25T19:48:36.351 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-03-25T19:48:40.338 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:48:40.352 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:48:40.448 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-25T19:48:40.461 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-03-25T19:48:44.453 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:48:44.462 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:48:44.564 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-03-25T19:48:44.571 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-03-25T19:48:44.665 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-03-25T19:48:44.672 INFO:teuthology.orchestra.console.smithi195:Power off completed 2024-03-25T19:48:44.777 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 4958.792445421219 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=985edf2b50264708b0e96dd52bc1e546 status: fail success: false 2024-03-25T19:48:44.777 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:48:44.858 INFO:teuthology.run:FAIL